[jira] [Commented] (CASSANDRA-8586) support millions of sstables by lazily acquiring/caching/dropping filehandles

["Benedict (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272935#comment-14272935 ] 

Benedict commented on CASSANDRA-8586:
-------------------------------------

This is only one possible problem. Another is the amount of bumpf we need to maintain each sstable in memory and, in particular, manage their lifecycles. Right now, for any modification of the sstable set, we copy our collection of sstables, update and then replace. With millions of sstables, this will become a significant expense and source of garbage. The interval tree is likely to be an even worse example of the same kind of behaviour (but also unnecessary in this scenario, so could perhaps be switched off). It is likely there are other similar problems.

> support millions of sstables by lazily acquiring/caching/dropping filehandles
> -----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8586
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8586
>             Project: Cassandra
>          Issue Type: New Feature
>            Reporter: Tupshin Harper
>            Assignee: Aleksey Yeschenko
>             Fix For: 3.1
>
>
> This might turn into a meta ticket if other obstacles are found in the goal of supporting a huge number of sstables.
> Technically, the only gap that I know of to prevent us from supporting absurd numbers of sstables is the fact that we hold on to an open filehandle for every single sstable. 
> For use cases that are willing to take a hit to read-performance in order to achieve high densities and low write amplification, a mechanism for only retaining file handles for recently read sstables could be very valuable.
> This will allow for alternate compaction strategies and compaction strategy tuning that don't try to optimize for read performance as aggresively.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)