[jira] [Reopened] (CASSANDRA-5555) Allow sstableloader to handle a larger number of files

["Yuki Morishita (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-5555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yuki Morishita reopened CASSANDRA-5555:
---------------------------------------


This fix can send wrong "estimated number of keys" for creating BF on the streamed node, since calculating estimate uses index summary.

My proposed fix is to make index summary completely optional. That is, when Summary.db file is present, load that and use it. We also add an option not to load Summary.db. And when the file is not present nor the user choose not to load the summary, we just scan sequentially on index file(Index.db) for "estimated number of keys".


                
> Allow sstableloader to handle a larger number of files
> ------------------------------------------------------
>
>                 Key: CASSANDRA-5555
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5555
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core, Tools
>            Reporter: Tyler Hobbs
>            Assignee: Jonathan Ellis
>             Fix For: 1.2.6
>
>         Attachments: 5555-01.txt, 5555-02.txt, CASSANDRA-5555.txt, CASSANDRA-5555.txt
>
>
> With the default heap size, sstableloader will OOM when there are roughly 25k files in the directory to load.  It's easy to reach this number of files in a single LCS column family.
> By avoiding creating all SSTableReaders up front in SSTableLoader, we should be able to increase the number of files that sstableloader can handle considerably.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira