[jira] [Created] (KAFKA-1404) Close unused log file

["Xinyao Hu (JIRA)" <ji...@apache.org>]

Xinyao Hu created KAFKA-1404:
--------------------------------

             Summary: Close unused log file
                 Key: KAFKA-1404
                 URL: https://issues.apache.org/jira/browse/KAFKA-1404
             Project: Kafka
          Issue Type: Improvement
          Components: core
    Affects Versions: 0.8.1
            Reporter: Xinyao Hu
            Priority: Critical


This is some what related to KAFKA-1403. 

One way to hack KAFKA-1403 is to roll a new file in a short period of time. However, this will result in much more file descriptor open. Take our application for example, each server hosts about 5k topic-partition, if we roll a new file per hour, we will have over 100k file descriptors open (I checked only .log is open but not .index which might be pinned in memory). 

In most of the application, very few of these file descriptors will be read. We should provide a parameter like max.num.fds and do LRU inside kafka to decide which fds should be open. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)