You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Jonathan <jo...@gmail.com> on 2011/10/05 19:00:43 UTC
Issues with .tmp files in HDFS
Hey experts,
I have flume writing to a directory in hdfs. I then fire off a cron job to
move that data into hive every five minutes. The problem that I am having is
that the .tmp files are also moved and start causing errors on the collector
that is writing the files to hdfs. Is there any way to get rid of the .tmp
files or to have them in a different directory then the other files? Any
other suggestions on how I can work around this issue?
Jonathan
Re: Issues with .tmp files in HDFS
Posted by Eric Sammer <es...@cloudera.com>.
Jonathan:
You can also use a path filter.
http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/fs/FileSystem.html#listStatus(org.apache.hadoop.fs.Path,
org.apache.hadoop.fs.PathFilter)
In fact, you should, otherwise you could be moving files you aren't aware of
(like tmp files).
On Wed, Oct 5, 2011 at 10:00 AM, Jonathan <jo...@gmail.com> wrote:
> Hey experts,
>
> I have flume writing to a directory in hdfs. I then fire off a cron job to
> move that data into hive every five minutes. The problem that I am having is
> that the .tmp files are also moved and start causing errors on the collector
> that is writing the files to hdfs. Is there any way to get rid of the .tmp
> files or to have them in a different directory then the other files? Any
> other suggestions on how I can work around this issue?
>
> Jonathan
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Issues with .tmp files in HDFS
Posted by Chetan Sarva <cs...@evidon.com>.
I'm not sure why, but we don't get any .tmp files when writing to HDFS.
Which version of Flume are you using? Either way, you could do something
like this:
hadoop fs -ls /collect/ | awk '{print $8}' | egrep -v tmp$ | xargs -in 1
hadoop fs -mv {} /hive/
might be a bit slower but it should work..
On Wed, Oct 5, 2011 at 1:00 PM, Jonathan <jo...@gmail.com> wrote:
> Hey experts,
>
> I have flume writing to a directory in hdfs. I then fire off a cron job to
> move that data into hive every five minutes. The problem that I am having is
> that the .tmp files are also moved and start causing errors on the collector
> that is writing the files to hdfs. Is there any way to get rid of the .tmp
> files or to have them in a different directory then the other files? Any
> other suggestions on how I can work around this issue?
>
> Jonathan
>