You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Lenin Raj <em...@gmail.com> on 2013/10/25 22:43:06 UTC

Flume is not adding .tmp suffix for in-progress files

Hi All,

I am having a simple agent config like below:

agent.sources = Twitter
agent.channels = MemChannel
agent.sinks = FileSink

agent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
agent.sources.Twitter.channels = MemChannel

agent.sinks.FileSink.channel = MemChannel
agent.sinks.FileSink.type = file_roll
agent.sinks.FileSink.sink.directory = /var/data/
agent.sinks.FileSink.sink.rollInterval = 60

agent.channels.MemChannel.type = memory
agent.channels.MemChannel.capacity = 50000
agent.channels.MemChannel.transactionCapacity = 1000


root@sys:/var/data# ls -ltrh
total 8.0K
-rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
-rw-r--r-- 1 root root    0 Oct 26 02:01 1382733013173-2
root@ sys:/var/data# ls -ltrh
total 12K
-rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
-rw-r--r-- 1 root root 2.1K Oct 26 02:01 1382733013173-2

As you can see, the file "1382733013173-2" is being modified without the
".tmp" prefix.

My Flume version:
Flume 1.4.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Does the config file need any modifications to get .tmp suffix? I need this
suffix to exclude this file from my file watcher script.

Thanks,
Lenin

Re: Flume is not adding .tmp suffix for in-progress files

Posted by Hari Shreedharan <hs...@cloudera.com>.
“.tmp” is added only for the HDFS sink, not the rolling file sink.  


Thanks,
Hari


On Friday, October 25, 2013 at 1:43 PM, Lenin Raj wrote:

> Hi All,
>  
> I am having a simple agent config like below:
>  
> agent.sources = Twitter
> agent.channels = MemChannel
> agent.sinks = FileSink
>  
> agent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
> agent.sources.Twitter.channels = MemChannel
>  
> agent.sinks.FileSink.channel = MemChannel
> agent.sinks.FileSink.type = file_roll
> agent.sinks.FileSink.sink.directory = /var/data/
> agent.sinks.FileSink.sink.rollInterval = 60
>  
> agent.channels.MemChannel.type = memory
> agent.channels.MemChannel.capacity = 50000
> agent.channels.MemChannel.transactionCapacity = 1000
>  
>  
> root@sys:/var/data# ls -ltrh
> total 8.0K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root    0 Oct 26 02:01 1382733013173-2
> root@ sys:/var/data# ls -ltrh
> total 12K
> -rw-r--r-- 1 root root 4.9K Oct 26 02:00 1382733013173-1
> -rw-r--r-- 1 root root 2.1K Oct 26 02:01 1382733013173-2
>  
> As you can see, the file "1382733013173-2" is being modified without the ".tmp" prefix.
>  
> My Flume version:
> Flume 1.4.0
> Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
>  
>  
> Does the config file need any modifications to get .tmp suffix? I need this suffix to exclude this file from my file watcher script.
>  
> Thanks,
> Lenin