You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2011/09/08 22:44:09 UTC
[jira] [Created] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
SlabAllocator OOMs much faster than HeapAllocator
-------------------------------------------------
Key: CASSANDRA-3163
URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0
Environment: 3 nodes, 1G heap, RF=2
Reporter: Brandon Williams
Fix For: 1.0
SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-3163:
----------------------------------------
Attachment: system.log.gz
Here's a system.log with your extra debug statements and Memtable set to DEBUG. It OOM'd many times, but never actually dumped.
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0
>
> Attachments: system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3163:
--------------------------------------
Attachment: (was: 3163-v3.txt)
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3163-v2.txt, 3163-v3.txt, 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3163.
---------------------------------------
Resolution: Fixed
Reviewer: brandon.williams
Assignee: Jonathan Ellis
committed
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 3163-v2.txt, 3163-v3.txt, 3163-v4.txt, 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3163:
--------------------------------------
Attachment: 3163-v2.txt
v2 re-clones into indexposition instead of trying to avoid cloning at all. this should be a small penalty since we're only re-cloning sampled keys, not all of them.
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0
>
> Attachments: 3163-v2.txt, 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100680#comment-13100680 ]
Jonathan Ellis commented on CASSANDRA-3163:
-------------------------------------------
what liveRatio was it logging on the OOM run?
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3163:
--------------------------------------
Attachment: 3163-v4.txt
v4 fixes NPE on sstable load
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3163-v2.txt, 3163-v3.txt, 3163-v4.txt, 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101406#comment-13101406 ]
Brandon Williams commented on CASSANDRA-3163:
---------------------------------------------
+1
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3163-v2.txt, 3163-v3.txt, 3163-v4.txt, 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3163:
--------------------------------------
Attachment: 3163-v3.txt
v3 also applies getMinimalKey to SSTR.first, last fields.
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3163-v2.txt, 3163-v3.txt, 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3163:
--------------------------------------
Attachment: 3163.txt
What is happening is, the slab-allocated row keys are being put into the IndexSummary of the new sstable on flush. So the regions can only be GC's post-compaction.
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0
>
> Attachments: 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3163) SlabAllocator OOMs much faster
than HeapAllocator
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3163:
--------------------------------------
Attachment: 3163-v3.txt
> SlabAllocator OOMs much faster than HeapAllocator
> -------------------------------------------------
>
> Key: CASSANDRA-3163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: 3 nodes, 1G heap, RF=2
> Reporter: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3163-v2.txt, 3163-v3.txt, 3163.txt, system.log.gz
>
>
> SlabAllocator will OOM with stress at around 5.5M rows, which in this configuration is roughly 3.3M rows per node. Reverting to the HeapAllocator allowed all 10M rows to finish. Examining the SA heap dump in MAT just shows ~98% of the heap is in 'remainder'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira