Flume checking for new files in a direcotry

[Jonathan <jo...@gmail.com>]

Hi,

I have been trying to find documentation on this use case but
I haven't found any yet. Is there a way to get flume to see when new files
are added to the directory instead of waiting for a file append? I am trying
to use flume with custom log files that are written to files at a set
interval. Would flume delete the files after it has processed them?

On a separate note is there a link where I can view the email history of
this group?

Thanks
Jonathan

Re: Flume checking for new files in a direcotry

[Mingjie Lai <mj...@gmail.com>]

> Is there a way to get flume to see when new
> files are added to the directory instead of waiting for a file append?

Don't know your exactly use case. Have you tried tailDir source? I saw 
it from the user guide but didn't try it.

 > On a separate note is there a link where I can view the email history of
 > this group?

It used to be on google group at:
https://groups.google.com/a/cloudera.org/group/flume-user/topics

It just moved to apache recently:
http://mail-archives.apache.org/mod_mbox/incubator-flume-user/

-mingjie

On 08/19/2011 12:06 PM, Jonathan wrote:
> Hi,
>
> I have been trying to find documentation on this use case but
> I haven't found any yet. Is there a way to get flume to see when new
> files are added to the directory instead of waiting for a file append? I
> am trying to use flume with custom log files that are written to files
> at a set interval. Would flume delete the files after it has processed them?
>
> On a separate note is there a link where I can view the email history of
> this group?
>
> Thanks
> Jonathan