[jira] [Created] (FLUME-2648) File Channel restarts can fail if metadata file creation fails during file roll

["Hari Shreedharan (JIRA)" <ji...@apache.org>]

Hari Shreedharan created FLUME-2648:
---------------------------------------

             Summary: File Channel restarts can fail if metadata file creation fails during file roll
                 Key: FLUME-2648
                 URL: https://issues.apache.org/jira/browse/FLUME-2648
             Project: Flume
          Issue Type: Bug
            Reporter: Hari Shreedharan
            Assignee: Hari Shreedharan


If during the creation of a new File in the data dirs, the creation of the metadata file fails - then the log-n file is simply left as an orphan. During replay though, this causes issues - as we try to read the metadata file at this time, causing the replay to fail and the file channel being unable to restart. 

To fix this we must remove the data file if the creation of the metadata file fails. Or we could simply ignore files whose metadata files don't exist (though in this case, user error could cause data loss). If the delete fails, then there would a problem though. So if the metafile is missing during replay, we must still fail - though more gracefully, and ask the operator to remove the corresponding data file. It might actually be possible to figure out if the data file has any content at all - by checking if size == 0 (need to verify this) - in that case we can probably remove/ignore that file.

I will submit a patch for this soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)