You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by "Krishna Kumar (JIRA)" <ji...@apache.org> on 2014/02/24 16:43:19 UTC

[jira] [Created] (FLUME-2331) Large TMP files created and never closed

Krishna Kumar created FLUME-2331:
------------------------------------

             Summary: Large TMP files created and never closed
                 Key: FLUME-2331
                 URL: https://issues.apache.org/jira/browse/FLUME-2331
             Project: Flume
          Issue Type: Bug
          Components: File Channel
    Affects Versions: v1.4.0
            Reporter: Krishna Kumar


We are currently writing files to Hadoop partitioned by year, month, day via Flume. File rollovers are done every 5 minutes. Recently, we noticed that this file rollover stops happening sometime during the day and that further data is written to an open TMP file. Because there are no further file rollovers, this TMP file becomes very large. At the end of the day, the TMP is not closed also and Flume goes to the next day, creating new files.

We use a "." prefix to prevent Hive from complaining of the open TMP file. Because of this issue where the TMP file is never closed, the file remains hidden to Hive even after the day ends.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)