Flume-ng (v1.3.0) multi tail or tailing directory

[Dibyajyoti Ghosh <di...@gmail.com>]

Hi flume-dev,

In flume v1.3.0 bundled with cdh-4.2.0 I am aware of exec source where one
can do 'tail -f' for a single mutable file.

Spool-dir is available but files have to be uniquely named and immutable.

Is there some way I can tail a directory much like multi tail / tailDir
option in the current flume version?

thanks a lot,
- dib

Re: Flume-ng (v1.3.0) multi tail or tailing directory

[Dibyajyoti Ghosh <di...@gmail.com>]

Hi,

I understand that tail can be applied to multiple files at once but that
will combine all output together and my intension is to keep log data from
different files separate from each other.

Another way is to treat each file to monitor as distinct sources and then
write output to distinct channels (file / memory) and shipping them to
different sinks.

agent.sources = s1 s2 ... sn
agent.channels = fc1 fc2 ... fcn
agent.sinks = sk1 sk2 ... sn

agent.sources.s1.exec  = tail -F -n 0 /var/log/myapp1.log
agent.sources.s2.exec  = tail -F -n 0 /var/log/myapp2.log
.
.
agent.sources.sn.exec  = tail -F -n 0 /var/log/myappn.log

Now s1 ---> fc1 ---> sk1
s2 ---> fc2 ---> sk2
s3 ---> fc3 ---> sk3
..
..
etc.

I am looking for a generic way of achieving the same as explained above if
possible.

Thanks,
- dib


On Thu, Mar 21, 2013 at 2:51 PM, Dibyajyoti Ghosh <dibyajyotighosh@gmail.com
> wrote:

> Hi flume-dev,
>
> In flume v1.3.0 bundled with cdh-4.2.0 I am aware of exec source where one
> can do 'tail -f' for a single mutable file.
>
> Spool-dir is available but files have to be uniquely named and immutable.
>
> Is there some way I can tail a directory much like multi tail / tailDir
> option in the current flume version?
>
> thanks a lot,
> - dib
>



On Thu, Mar 21, 2013 at 2:51 PM, Dibyajyoti Ghosh <dibyajyotighosh@gmail.com
> wrote:

> Hi flume-dev,
>
> In flume v1.3.0 bundled with cdh-4.2.0 I am aware of exec source where one
> can do 'tail -f' for a single mutable file.
>
> Spool-dir is available but files have to be uniquely named and immutable.
>
> Is there some way I can tail a directory much like multi tail / tailDir
> option in the current flume version?
>
> thanks a lot,
> - dib
>