You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2011/09/09 21:48:09 UTC
[jira] [Created] (CASSANDRA-3171) AbstractCompactionIterable uses a
1MB buffer for every sstable
AbstractCompactionIterable uses a 1MB buffer for every sstable
--------------------------------------------------------------
Key: CASSANDRA-3171
URL: https://issues.apache.org/jira/browse/CASSANDRA-3171
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.0
Reporter: Brandon Williams
Assignee: Brandon Williams
Fix For: 1.0.0
This causes an OOM for any compaction task that needs to open all the sstables when you have a lot of them (repair, major, etc)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3171) AbstractCompactionIterable uses a
1MB buffer for every sstable
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-3171:
----------------------------------------
Attachment: 3171.txt
Patch to drop buffer size to 64k, as is the RandomAccessReader default.
> AbstractCompactionIterable uses a 1MB buffer for every sstable
> --------------------------------------------------------------
>
> Key: CASSANDRA-3171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3171
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3171.txt
>
>
> This causes an OOM for any compaction task that needs to open all the sstables when you have a lot of them (repair, major, etc)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3171) AbstractCompactionIterable uses
a 1MB buffer for every sstable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101524#comment-13101524 ]
Jonathan Ellis commented on CASSANDRA-3171:
-------------------------------------------
Is there anywhere we actually need non-default buffer sizes? I think we should probably rip out that constructor parameter (on both RAR and SW) to force everyone to be sane whether they like it or not.
See also: the 10MB buffer in streaming and the 8MB one in IndexWriter.
> AbstractCompactionIterable uses a 1MB buffer for every sstable
> --------------------------------------------------------------
>
> Key: CASSANDRA-3171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3171
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3171.txt
>
>
> This causes an OOM for any compaction task that needs to open all the sstables when you have a lot of them (repair, major, etc)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3171) AbstractCompactionIterable uses a
1MB buffer for every sstable
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-3171:
----------------------------------------
Reviewer: jbellis
> AbstractCompactionIterable uses a 1MB buffer for every sstable
> --------------------------------------------------------------
>
> Key: CASSANDRA-3171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3171
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3171.txt
>
>
> This causes an OOM for any compaction task that needs to open all the sstables when you have a lot of them (repair, major, etc)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3171) AbstractCompactionIterable uses
a 1MB buffer for every sstable
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101570#comment-13101570 ]
Brandon Williams commented on CASSANDRA-3171:
---------------------------------------------
+1
> AbstractCompactionIterable uses a 1MB buffer for every sstable
> --------------------------------------------------------------
>
> Key: CASSANDRA-3171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3171
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3171-v2.txt, 3171.txt
>
>
> This causes an OOM for any compaction task that needs to open all the sstables when you have a lot of them (repair, major, etc)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3171) AbstractCompactionIterable uses
a 1MB buffer for every sstable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3171.
---------------------------------------
Resolution: Fixed
committed
> AbstractCompactionIterable uses a 1MB buffer for every sstable
> --------------------------------------------------------------
>
> Key: CASSANDRA-3171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3171
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3171-v2.txt, 3171.txt
>
>
> This causes an OOM for any compaction task that needs to open all the sstables when you have a lot of them (repair, major, etc)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3171) AbstractCompactionIterable uses a
1MB buffer for every sstable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3171:
--------------------------------------
Attachment: 3171-v2.txt
v2 removes unnecessary bufferSize parameters across the call heirarchy. Main exception is CRAR, to which I added
{code}
// TODO refactor this to separate concept of "buffer to avoid lots of read() syscalls" and "compression buffer"
{code}
> AbstractCompactionIterable uses a 1MB buffer for every sstable
> --------------------------------------------------------------
>
> Key: CASSANDRA-3171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3171
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 1.0.0
>
> Attachments: 3171-v2.txt, 3171.txt
>
>
> This causes an OOM for any compaction task that needs to open all the sstables when you have a lot of them (repair, major, etc)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira