[jira] [Commented] (KUDU-1399) LBM fd usage sometimes doesn't scale right

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311993#comment-15311993 ] 

Todd Lipcon commented on KUDU-1399:
-----------------------------------

8d377ac0fe2be0f9176526e9db745ea6dd70ce86 (KUDU-678) might have helped this a bit, but it's still probably worth implementing some kind of compaction which relocates blocks out of containers that only have <5% full or somesuch

> LBM fd usage sometimes doesn't scale right
> ------------------------------------------
>
>                 Key: KUDU-1399
>                 URL: https://issues.apache.org/jira/browse/KUDU-1399
>             Project: Kudu
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jean-Daniel Cryans
>            Assignee: Todd Lipcon
>
> Noticed on a cluster that runs YCSB non-stop in a loop, using a new table each time and deleting an old one, while ensuring that it keeps at least 10 tables to simulate a "loaded" cluster.
> After a few weeks of running, the cluster's file descriptor usage started triggering alerts in the monitoring system. The total data consumption never really changes since we keep at most 10 tables. Todd found that we have thousands of really small log block manager containers.
> The workaround at the moment is to simply increase the ulimit for open file descriptors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)