You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/09/21 01:48:09 UTC
[jira] [Created] (CASSANDRA-3234) LeveledCompaction has several
performance problems
LeveledCompaction has several performance problems
--------------------------------------------------
Key: CASSANDRA-3234
URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.0
Reporter: Jonathan Ellis
Assignee: Jonathan Ellis
Fix For: 1.0.0
Two main problems:
- BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
- ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3234:
----------------------------------------
Attachment: 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch
The problem is that now CFS.removeDeleted throws a ConcurrentModificationException. Attaching v4 that replaces the
{noformat}
for (IColumn c : cf)
cf.remove(c.name())
{noformat}
pattern by
{noformat}
Iterator<IColumn> iter = cf.iterator()
while (iter.hasNext())
{
IColumn c = iter.next();
iter.remove();
}
{noformat}
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0002-fix-leveled-BF-size-calculation.txt
0001-optimize-single-source-case-for-MergeIterator.txt
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-fix-leveled-BF-size-calculation.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113272#comment-13113272 ]
Sylvain Lebresne commented on CASSANDRA-3234:
---------------------------------------------
The patch as is doesn't compile because it removes the import of StringUtils in LeveledManifest.java (it also add the import of DataTracker in AbstractCompactionStrategy which I think is not useful). But other than +1 on 06 v3.
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: (was: 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch)
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3234:
----------------------------------------
Attachment: 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch
I haven't looked at the 3 first patches, but on patch 4 and 5.
+1 on patch 4 (though I agree with the comment in there that it's not the more beautiful refactor ever :))
On patch 5, it cloneMeShallow the first read column family and basically skip all the columns, so that's wrong. Attaching a v2 that makes SSTII directly use the right ISortedColumn factory (to avoid full cloning). Problem is this doesn't translate to ParallelCompactionIterable too well since the actual read is deep into the code. For it, I think we have 2 easy solutions:
* Just use ArraySortedColumns all the way. This is actually ok because addAll works whatever the input is, it does a merge.
* Do a full clone to a TreeMapBacked CF on the first cf read
* Use TreeMapBack CFs all the way.
I went with the first solution in the patch attached (more because it requires the less changes than anything else), though that's probably not optimal for LeveledCompaction (but I'm not sure ParallelCompaction is useful for LeveledCompaction).
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109702#comment-13109702 ]
Jonathan Ellis commented on CASSANDRA-3234:
-------------------------------------------
bq. that's probably not optimal for LeveledCompaction
Can you elaborate?
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0003-fix-leveled-BF-size-calculation.txt
0002-add-TrivialOneToOne-optimization.txt
0001-optimize-single-source-case-for-MergeIterator.txt
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: (was: 0002-add-TrivialOneToOne-optimization.txt)
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112767#comment-13112767 ]
Brandon Williams commented on CASSANDRA-3234:
---------------------------------------------
+1
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt
simple 05 v3 attached using just ABSC
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: (was: 0002-fix-leveled-BF-size-calculation.txt)
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
updated to "Sets.difference(L0, sstablesToIgnore).size() + manifest.getLevelCount()"
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: (was: 0001-optimize-single-source-case-for-MergeIterator.txt)
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
updated 06
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109710#comment-13109710 ]
Jonathan Ellis commented on CASSANDRA-3234:
-------------------------------------------
{code}
+ thisCF = row.getColumnFamilyWithColumns(cf == null ? TreeMapBackedSortedColumns.factory() : ArrayBackedSortedColumns.factory());
{code}
I don't understand why not just ABSC here.
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: (was: 0001-optimize-single-source-case-for-MergeIterator.txt)
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
06 attached as well.
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109727#comment-13109727 ]
Sylvain Lebresne commented on CASSANDRA-3234:
---------------------------------------------
ABSC.addAll does a merge. So if you do cf1.addAll(cf2) and cf1 and cf2 are comparable in size, that's likely as good as it gets. However, if cf2 has much less columns than cf1, then it's likely that TMBSC.addAll (that adds the columns of cf2 one by one) will be faster.
So I guess it all depends on how many sstables we are merging. If we have a lot of them, then adding to TMBSC will be a win (my claim on LeveledCompaction was misdirected, I was thinking of it as lots of sstables, but that doesn't mean we'll merge lots of sstables (unless L0 is behind)). Anyway, it can very well be that ABSC is better in the vast majority of cases so I'm fine with using just that.
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109244#comment-13109244 ]
Jonathan Ellis commented on CASSANDRA-3234:
-------------------------------------------
split OneToOne classes into separate patches and fixed bug in estimated tables calculation
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
v2 improves leveled expensiveness equation to
L0.size() + count(levels) - ignore.size()
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v2.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction-v3.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction-v4.patch, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch, 0006-avoid-echoedRow-when-checking-shouldPurge-is-more-ex.patch
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
0004-avoid-calling-shouldPurge-unless-necessary.txt
0003-fix-leveled-BF-size-calculation.txt
0002-add-TrivialOneToOne-optimization.txt
0001-optimize-single-source-case-for-MergeIterator.txt
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
> Attachments: 0001-optimize-single-source-case-for-MergeIterator.txt, 0002-add-TrivialOneToOne-optimization.txt, 0003-fix-leveled-BF-size-calculation.txt, 0004-avoid-calling-shouldPurge-unless-necessary.txt, 0005-use-Array-and-Tree-backed-columns-in-compaction.txt
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3234) LeveledCompaction has several
performance problems
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3234:
--------------------------------------
Attachment: (was: 0003-fix-leveled-BF-size-calculation.txt)
> LeveledCompaction has several performance problems
> --------------------------------------------------
>
> Key: CASSANDRA-3234
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3234
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Jonathan Ellis
> Assignee: Jonathan Ellis
> Fix For: 1.0.0
>
>
> Two main problems:
> - BF size calculation doesn't take into account LCS breaking the output apart into "bite sized" sstables, so memory use is much higher than predicted
> - ManyToMany merging is slow. At least part of this is from running the full reducer machinery against single input sources, which can be optimized away.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira