You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org> on 2012/11/26 23:21:00 UTC

[jira] [Assigned] (FLUME-1702) HDFSEventSink should write to a hidden file as opposed to a .tmp file

     [ https://issues.apache.org/jira/browse/FLUME-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jarek Jarcec Cecho reassigned FLUME-1702:
-----------------------------------------

    Assignee: Jarek Jarcec Cecho
    
> HDFSEventSink should write to a hidden file as opposed to a .tmp file
> ---------------------------------------------------------------------
>
>                 Key: FLUME-1702
>                 URL: https://issues.apache.org/jira/browse/FLUME-1702
>             Project: Flume
>          Issue Type: Improvement
>            Reporter: Brock Noland
>            Assignee: Jarek Jarcec Cecho
>
> Currently we write to a .tmp file. The problem is that if MR jobs are being run on the directory we are writing to, then it's common for an MR job to list the directory, get a .tmp file and then in the mean time the .tmp file is renamed causing the job to fail when run.
> Using JavaMR you can use a PathFilter to avoid this, however a custom solution is required for Pig, Hive, etc.
> Perhaps we should write to a hidden file so that MR never tries to process data in flight.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira