[jira] [Created] (FLUME-2979) File descriptor leaks in TaildirSource

["Kun Liu (JIRA)" <ji...@apache.org>]

Kun Liu created FLUME-2979:
------------------------------

             Summary: File descriptor leaks in TaildirSource
                 Key: FLUME-2979
                 URL: https://issues.apache.org/jira/browse/FLUME-2979
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: v1.7.0
            Reporter: Kun Liu


TaildirSource creates ReliableTaildirEventReader object in start() method,  and the constructor of ReliableTaildirEventReader will try to open matched files in updateTailFiles(). However, If the flume process has no permission to some matched files, then the updateTailFiles() method throws IOException and interrupts the creation of ReliableTaildirEventReader object. The following logs show the opening of some matched files and the exception thrown when try to open a file with no permission.

------------logs--------------
25 Aug 2016 12:50:28,337 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294)  - Opening file: /home/work/quota/test.2016-06-27-121958, inode: 91668538, pos: 0
25 Aug 2016 12:50:28,337 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294)  - Opening file: /home/work/quota/test.2016-06-27-111458, inode: 91668558, pos: 0
25 Aug 2016 12:50:28,337 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294)  - Opening file: /home/work/quota/test.2016-06-27-080458, inode: 91668536, pos: 0
25 Aug 2016 12:50:28,338 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile:294)  - Opening file: /home/work/quota/test.2016-06-27-140958, inode: 91668655, pos: 0
25 Aug 2016 12:50:28,338 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253)  - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: /opt/bls/.bls/flume/taildir_position/6d124945-0f73-42ed-7178-56a2168fc5f5/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
org.apache.flume.FlumeException: Failed opening file: /home/work/quota/test.2016-06-27-140958
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:297)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:260)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:94)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.<init>(ReliableTaildirEventReader.java:48)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:361)
        at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:100)
        at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:72)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: /home/work/quota/test.2016-06-27-140958 (Permission denied)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(Unknown Source)
        at org.apache.flume.source.taildir.TailFile.<init>(TailFile.java:60)
        at org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:295)
        ... 14 more
------------logs--------------


As the creation of ReliableTaildirEventReader is interrupted, the TaildirSource catches the IOException and throws FlumeException in start() method, but the opened files are not closed, which leads to the leak of file descriptors. As time goes by,  the PollableSourceRunner keeps starting TaildirSource, and the number of opened files in flume process exceeds the upper limit, then the 'Too many open files' error occurs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)