[jira] [Commented] (CASSANDRA-5555) Allow sstableloader to handle a larger number of files

["Jonathan Ellis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-5555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13666324#comment-13666324 ] 

Jonathan Ellis commented on CASSANDRA-5555:
-------------------------------------------

You're right, it's a bit messy. :)

Digging in a bit, I'm actually a bit confused by the premise here -- unless you're generating sstables to load on a machine with vastly more disk space than your cluster members, you should be able to open the sstables you're generating, since that's what the nodes you're streaming to will have to do.

(Before looking at the code, I assumed we were opening some kind of scanner or buffer per sstable a la CASSANDRA-4142 but it looks like it's just the SSTableReader.)

bq. With the default heap size, sstableloader will OOM when there are roughly 25k files in the directory to load

This default?

{code}
$JAVA -ea -cp $CLASSPATH -Xmx256M
{code}

Honestly I think telling people "increase the heap size" is a perfectly adequate solution if that's really what we're running with right now.  Increasing it to 1GB out of the box also seems reasonable.
                
> Allow sstableloader to handle a larger number of files
> ------------------------------------------------------
>
>                 Key: CASSANDRA-5555
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5555
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Tyler Hobbs
>            Assignee: Dave Brosius
>             Fix For: 1.2.6
>
>         Attachments: 5555-01.txt
>
>
> With the default heap size, sstableloader will OOM when there are roughly 25k files in the directory to load.  It's easy to reach this number of files in a single LCS column family.
> By avoiding creating all SSTableReaders up front in SSTableLoader, we should be able to increase the number of files that sstableloader can handle considerably.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira