[jira] [Created] (ACCUMULO-416) reevaluate limiting the number of open files given HDFS improvements

["Adam Fuchs (Created) (JIRA)" <ji...@apache.org>]

reevaluate limiting the number of open files given HDFS improvements
--------------------------------------------------------------------

                 Key: ACCUMULO-416
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-416
             Project: Accumulo
          Issue Type: Improvement
          Components: tserver
            Reporter: Adam Fuchs
            Assignee: Keith Turner


Tablet servers limit the number of files that can be opened for scans and for major compactions. The two main reasons for this limit was to reduce our impact on HDFS, primarily regarding connections to data nodes, and to limit our memory usage related to preloading file indexes. A third reason might be that disk thrashing could become a problem if we try to read from too many places at once.

Two improvements may have made (or may soon make) this limit obsolete: HDFS now pools connections, and RFile now uses a multi-level index. With these improvements, is it reasonable to lift some of our open file restrictions? The tradeoff on query side might be availability vs. overall resource usage. On the compaction side, the tradeoff is probably write replication vs. thrashing on reads. I think we can make an argument that queries should be available at almost any cost, but the compaction tradeoff is not as clear. We should test the efficiency of compacting a large number of files to get a better feeling for how the two extremes effect read and write performance across the system.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira