[jira] [Resolved] (KUDU-1399) LBM fd usage sometimes doesn't scale right

["Adar Dembo (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KUDU-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Adar Dembo resolved KUDU-1399.
------------------------------
       Resolution: Fixed
    Fix Version/s: 1.2.0

As of commits 6a124f0 and 3fa9db7, the block managers now use an LRU cache to manage the fd count.

Going to close this one as fixed. If we decide we need to implement compaction for tiny containers, it'll be tackled later.


> LBM fd usage sometimes doesn't scale right
> ------------------------------------------
>
>                 Key: KUDU-1399
>                 URL: https://issues.apache.org/jira/browse/KUDU-1399
>             Project: Kudu
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jean-Daniel Cryans
>            Assignee: Adar Dembo
>             Fix For: 1.2.0
>
>
> Noticed on a cluster that runs YCSB non-stop in a loop, using a new table each time and deleting an old one, while ensuring that it keeps at least 10 tables to simulate a "loaded" cluster.
> After a few weeks of running, the cluster's file descriptor usage started triggering alerts in the monitoring system. The total data consumption never really changes since we keep at most 10 tables. Todd found that we have thousands of really small log block manager containers.
> The workaround at the moment is to simply increase the ulimit for open file descriptors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)