You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Lenin Raj <em...@gmail.com> on 2013/10/25 22:43:06 UTC
Flume is not adding .tmp suffix for in-progress files
Hi All,
I am having a simple agent config like below:
agent.sources = Twitter
agent.channels = MemChannel
agent.sinks = FileSink
agent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
agent.sources.Twitter.channels = MemChannel
agent.sinks.FileSink.channel = MemChannel
agent.sinks.FileSink.type = file_roll
agent.sinks.FileSink.sink.directory = /var/data/
agent.sinks.FileSink.sink.rollInterval = 60
agent.channels.MemChannel.type = memory
agent.channels.MemChannel.capacity = 50000
agent.channels.MemChannel.transactionCapacity = 1000
root@sys:/var/data# ls -ltrh
total 8.0K
-rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
-rw-r--r-- 1 root root 0 Oct 26 02:01 1382733013173-2
root@ sys:/var/data# ls -ltrh
total 12K
-rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
-rw-r--r-- 1 root root 2.1K Oct 26 02:01 1382733013173-2
As you can see, the file "1382733013173-2" is being modified without the
".tmp" prefix.
My Flume version:
Flume 1.4.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Does the config file need any modifications to get .tmp suffix? I need this
suffix to exclude this file from my file watcher script.
Thanks,
Lenin
Re: Flume is not adding .tmp suffix for in-progress files
Posted by Hari Shreedharan <hs...@cloudera.com>.
“.tmp” is added only for the HDFS sink, not the rolling file sink.
Thanks,
Hari
On Friday, October 25, 2013 at 1:43 PM, Lenin Raj wrote:
> Hi All,
>
> I am having a simple agent config like below:
>
> agent.sources = Twitter
> agent.channels = MemChannel
> agent.sinks = FileSink
>
> agent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
> agent.sources.Twitter.channels = MemChannel
>
> agent.sinks.FileSink.channel = MemChannel
> agent.sinks.FileSink.type = file_roll
> agent.sinks.FileSink.sink.directory = /var/data/
> agent.sinks.FileSink.sink.rollInterval = 60
>
> agent.channels.MemChannel.type = memory
> agent.channels.MemChannel.capacity = 50000
> agent.channels.MemChannel.transactionCapacity = 1000
>
>
> root@sys:/var/data# ls -ltrh
> total 8.0K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root 0 Oct 26 02:01 1382733013173-2
> root@ sys:/var/data# ls -ltrh
> total 12K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root 2.1K Oct 26 02:01 1382733013173-2
>
> As you can see, the file "1382733013173-2" is being modified without the ".tmp" prefix.
>
> My Flume version:
> Flume 1.4.0
> Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
>
>
> Does the config file need any modifications to get .tmp suffix? I need this suffix to exclude this file from my file watcher script.
>
> Thanks,
> Lenin