You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2011/01/11 22:36:48 UTC
[jira] Created: (CASSANDRA-1968) Increase JVM young generation size
Increase JVM young generation size
----------------------------------
Key: CASSANDRA-1968
URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
Project: Cassandra
Issue Type: Improvement
Affects Versions: 0.7.0
Reporter: Brandon Williams
Fix For: 0.7.1
Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-1968:
--------------------------------------
Attachment: 1968_v2.txt
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984580#action_12984580 ]
Jonathan Ellis commented on CASSANDRA-1968:
-------------------------------------------
Yes, I meant young gen.
An insert-heavy workload will have more of the young gen survive since we have to keep data around in memtables until flush. With a read-heavy load almost all of it is discarded. Here are some ParNew collections with about a 1.5G young gen, mTT=1, serving mostly reads:
{code}
49527.273: [GC 49527.274: [ParNew
Desired survivor size 107347968 bytes, new threshold 1 (max 1)
- age 1: 15782840 bytes, 15782840 total
: 1696530K->21789K(1887488K), 0.0737280 secs] 6279370K->4617936K(12373248K), 0.0745280 secs] [Times: user=0.41 sys=0.00, real=0.07 secs]
Total time for which application threads were stopped: 0.1053180 seconds
49534.739: [GC 49534.739: [ParNew
Desired survivor size 107347968 bytes, new threshold 1 (max 1)
- age 1: 14990096 bytes, 14990096 total
: 1699613K->18988K(1887488K), 0.0777830 secs] 6295760K->4629494K(12373248K), 0.0786270 secs]
{code}
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984445#action_12984445 ]
Todd Lipcon commented on CASSANDRA-1968:
----------------------------------------
What young gen collection rates are you seeing with 100M/core? Over in HBase land we've found young gen of that size to pause way too long.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-1968:
--------------------------------------
Attachment: 1968_v2.txt
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams resolved CASSANDRA-1968.
-----------------------------------------
Resolution: Fixed
Committed with the change from NewSize to Xmn
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984776#action_12984776 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
No, I'm saying the pause times during young-gen collections could be affected by old-gen because the pause includes doing the promotions to old-gen.
But, this may not be an issue depending on how the free lists are managed. If it's just "stick it in the right free list, promotion failure if it's full" then presumably this won't be a problem. I was mostly commenting that it's not implausible that there may be an issue that doesn't show up until after old-gen "maturity". Not making a positive claim, just sounding caution :)
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1968) Increase JVM young
generation size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984752#action_12984752 ]
Jonathan Ellis edited comment on CASSANDRA-1968 at 1/21/11 9:37 AM:
--------------------------------------------------------------------
Young gen collection always compacts (part of the reason it's a STW phase).
was (Author: jbellis):
Young gen compaction always compacts (part of the reason it's a STW phase).
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980426#action_12980426 ]
Jonathan Ellis commented on CASSANDRA-1968:
-------------------------------------------
bq. min(max_sensible_per_modern_cpu_core*num_cores, 1/4 heap size)
probably this, since new gen compaction is supposed to be parallelizable (not sure how much this is true in practice).
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980385#action_12980385 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
What young gen size did the GC converge on before your patch, in your particular test?
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-1968:
--------------------------------------
Attachment: (was: 1968_v2.txt)
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1968:
--------------------------------------
Remaining Estimate: 4h
Original Estimate: 4h
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-1968:
--------------------------------------
Attachment: 1968_v3.txt
(Same as v2 without accidentally left debug printout)
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984450#action_12984450 ]
Jonathan Ellis commented on CASSANDRA-1968:
-------------------------------------------
bq. What young gen collection rates are you seeing with 100M/core?
About 100ms for a 1G heap on 8 cores with a maxTenuringThreshold of 15. IIRC it's about half that with mTT=1.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1968:
----------------------------------------
Attachment: 1968.txt
Patch to automatically set the young generation to 1/4 the heap, which yields a 80-150% increase on both reads and writes in my testing.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984756#action_12984756 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
I know, but promoted objects have to be put into old-gen which is free-list based and subject to fragmentation. So on each young-gen collection you have a latency critical spike of promotions. To the extent that those promotions is a function of fragmentation, the pause times might be variable over time as a function of old space.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980371#action_12980371 ]
Brandon Williams commented on CASSANDRA-1968:
---------------------------------------------
Agreed, this needs further testing and tweaking. The patch was burning a hole in my laptop though, so I wanted to get it out there. :)
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984838#action_12984838 ]
Todd Lipcon commented on CASSANDRA-1968:
----------------------------------------
Sorry, I actually missed a step there - it first goes to the indexed free list for <16 words. If there's nothing available there, it goes to the sLAB. When these allocations are freed they go back into the indexed free list.
(at least this is my reading of the code, which I started looking at only yesterday, so would be great if someone else takes a look to confirm!)
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980446#action_12980446 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
Attaching a version that does that calculation. Not yet intended for direct inclusions; some of the numbers are arbitrary and maybe it should be cleaned up a bit.
Also, correcting myself on JDK 1.7: With CMS it's apparantly MaxGCMinorPauseMillis for young-gen (MaxGCPauseMillis). Very brief testing did not work well but probably because I failed to tweak some other setting to allow it to dynamically size properly. Nevermind, since 1.7 isn't out yet.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984611#action_12984611 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
Anyone got a good feel for how much pause times will be affected after long-time use? WIth a fixed new size, I'm a bit paranoid about pause times creeping up as young gen becomes more fragmented etc. However I'm not up enough on how CMS old-gen is managed to know whether this is an expected result.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984584#action_12984584 ]
Todd Lipcon commented on CASSANDRA-1968:
----------------------------------------
Cool, thanks. You may be interested in HBASE-3455 - probably applies to Cassandra as well. Feel free to jump in if you have thoughts.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982199#action_12982199 ]
Jonathan Ellis commented on CASSANDRA-1968:
-------------------------------------------
You want to use -Xmn rather than -XX:NewSize. The former sets a specific heap size while the latter only sets a lower bound, and the jvm will try to tune it from there -- which is usually not what we want.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12981459#action_12981459 ]
T Jake Luciani commented on CASSANDRA-1968:
-------------------------------------------
I tested this with Solandra and saw a 25% boost in write speed, read however was unchanged (though Solandra doesn't use thrift)
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984752#action_12984752 ]
Jonathan Ellis commented on CASSANDRA-1968:
-------------------------------------------
Young gen compaction always compacts (part of the reason it's a STW phase).
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983860#action_12983860 ]
Hudson commented on CASSANDRA-1968:
-----------------------------------
Integrated in Cassandra-0.7 #177 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/177/])
Increase JVM young generation size
Patch by Peter Schuller and brandonwilliams, reviewed by brandonwilliams
for CASSANDRA-1968
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-1968:
-------------------------------------------
Assignee: Brandon Williams
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-1968:
--------------------------------------
Comment: was deleted
(was: (Same as v2 without accidentally left debug printout))
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980364#action_12980364 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
Hmmm. The young gen will also directly affect the pause times for most collection pauses, as I'm sure you're aware. Depending on total heap size and depending on CPU count, 1/4 heap going to young gen might be huge and cause significant pause times.
It's certainly expected that larger young gens, up to a point, increase performance by avoiding promotion to old-gen, but should the out-of-the-box settings be so aggressive given the potential effects on latency?
How about something like min(max_sensible_default, 1/4 heap size) or perhaps min(max_sensible_per_modern_cpu_core*num_cores, 1/4 heap size)?
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984817#action_12984817 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
Thank you very much for that summary! Very insightful.
One question: Do you know what happens with those sLAB:s when the <16word objects need to be freed? Does an entire sLAB need to be free prior to reclamation?
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984778#action_12984778 ]
Todd Lipcon commented on CASSANDRA-1968:
----------------------------------------
The old gen allocation works like this:
- If it's smaller than 16 words, bump-the-pointer allocate into the "sLAB" - small linear allocation buffer. This buffer is allocated in chunks and then doled out for small objects
- If it's between 16 and 256 words, use "indexed free lists" - basically an array of freelists, one per size.
- If there are no objects free in a freelist, keep searching through bigger free lists until you find one. There is a "hint" pointer in each free list to allow you to skip directly to the next larger one that last succeeded in allocation, so it should be expected O(1)
- If the "indexed free lists" have no space that fit (or you're bigger than 256 words) then move to the BinaryTreeDictionary
- BTD is a balanced tree of chunks keyed by size. It uses a best fit strategy
- The chosen chunk is split in two, data structures updated, and the remaining half returned into the free space tracking (ie into indexed free lists if it's small, BTD if big)
So I would guess it's O(1) for small allocations, O(lg(number of fragments)) for large allocations.
There's also a concept of a PLAB (promotion local allocation buffer) which I don't quite understand yet - I think each collector thread in the young gen allocates a bigger chunk out of the old gen rather than allocating space for each object. This makes sense because the old gen allocations happen under one big lock.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984456#action_12984456 ]
Todd Lipcon commented on CASSANDRA-1968:
----------------------------------------
By 1G heap you mean 1G young gen? = 1G/second/core collection rate? How much data of the 1G young gen is live generally?
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12984766#action_12984766 ]
Jonathan Ellis commented on CASSANDRA-1968:
-------------------------------------------
old-gen STW compactions are going to be unpredictable, yes. that's pretty much unavoidable.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1968) Increase JVM young generation
size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980412#action_12980412 ]
Peter Schuller commented on CASSANDRA-1968:
-------------------------------------------
FWIW, on JDK 1.7, using MaxGCPauseMillis set to a reasonable value may be a good idea, allowing the VM to choose a young-gen size as large as possible to accomodate the pause time goal (while originally introduced for G1 it's also usable with CMS, though I haven't tried how well it works with CMS). Unfortunately this is not available before 1.7.
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1968) Increase JVM young generation size
Posted by "Peter Schuller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-1968:
--------------------------------------
Attachment: (was: 1968_v3.txt)
> Increase JVM young generation size
> ----------------------------------
>
> Key: CASSANDRA-1968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1968
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Attachments: 1968.txt, 1968_v2.txt
>
>
> Increasing the young generation size has a profound effect on performance and cpu utilization. We should set it explicitly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.