You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Dylan Jones (JIRA)" <ji...@apache.org> on 2015/04/15 14:55:59 UTC

[jira] [Commented] (FLUME-2458) Separate hdfs tmp directory for flume hdfs sink

    [ https://issues.apache.org/jira/browse/FLUME-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496138#comment-14496138 ] 

Dylan Jones commented on FLUME-2458:
------------------------------------

Hello,

Think I have found a workaround for now that allows this. If you overload the filePrefix with a path i.e.

hdfs.path = /flume
hdfs.filePrefix = data/
hdfs.fileSuffix = .avro
hdfs.inUsePrefix = tmp/

Real files: 
/flume/data/123456.avro 

Temp files: 
/flume/tmp/data/123456.avro.tmp 

You can then use the location /flume/data for external tables and the likes for map reduce jobs

> Separate hdfs tmp directory for flume hdfs sink
> -----------------------------------------------
>
>                 Key: FLUME-2458
>                 URL: https://issues.apache.org/jira/browse/FLUME-2458
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>    Affects Versions: v1.5.0.1
>            Reporter: Sverre Bakke
>            Assignee: Johny Rufus
>            Priority: Minor
>
> The current HDFS sink will write temporary files to the same directory as the final file will be stored. This is a problem for several reasons:
> 1) File moving
> When mapreduce fetches a list of files to be processed and then processes files that are then gone (i.e. are moved from .tmp to  whatever final name it is suppose to have), then the mapreduce job will crash.
> 2) File type
> When mapreduce decides how to process files, then it looks at files extension. If using compressed files, then it will decompress it for you. If the file has a .tmp file extension (in the same folder) then it will treat a compressed file as an uncompressed files, thus breaking the results of the mapreduce job.
> I propose that the sink gets an optional tmp path for storing these files to avoid these issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)