Fwd: Is it possible to get the file name from flume agent

[Junxian Yan <ju...@gmail.com>]

Copy this question to new group

---------- Forwarded message ----------
From: Junxian Yan <ju...@gmail.com>
Date: Tue, Jul 26, 2011 at 2:59 PM
Subject: Re: Is it possible to get the file name from flume agent
To: Flume Users <fl...@cloudera.org>


Or maybe a more specific description for my question:
Is it possible to get the original file name from flume agent output

R


On Tue, Jul 26, 2011 at 2:58 PM, Junxian Yan <ju...@gmail.com> wrote:

> Hi Guys
>
> I want to customize the collector output file name and the storing path by
> the log file name.
> Such as if the log name is a-b.log
> I want the storing path is like /a/b/a-b.log
>
> Is it possible?
>
>
>

Re: Fwd: Is it possible to get the file name from flume agent

[Dan Everton <da...@iocaine.org>]

You can get the original file name (if you're using any of the tail
sources) from the tailSrcFile attribute. We use it like this:

collectorSink("file:///logs/flume/apps/%{application}/access/%Y-%m-%d/%H/",
"%{tailSrcFile}-%{host}-", 300000),

The second argument is the file name that will be used. If you're not
using the tail sources, you could add a custom attribute to your flow
definition that contains the file name.

As for parsing the file name and using that to create the folder names,
this thread
(https://groups.google.com/a/cloudera.org/group/flume-user/msg/9e78675c759223ed)
has an example of using the regex extractor to pull parts of the message
out to use in the collector file name.

Cheers,
Dan 

On Wed, 27 Jul 2011 12:03 +0800, "Junxian Yan" <ju...@gmail.com>
wrote:
> Copy this question to new group
> 
> ---------- Forwarded message ----------
> From: Junxian Yan <ju...@gmail.com>
> Date: Tue, Jul 26, 2011 at 2:59 PM
> Subject: Re: Is it possible to get the file name from flume agent
> To: Flume Users <fl...@cloudera.org>
> 
> 
> Or maybe a more specific description for my question:
> Is it possible to get the original file name from flume agent output
> 
> R
> 
> 
> On Tue, Jul 26, 2011 at 2:58 PM, Junxian Yan <ju...@gmail.com>
> wrote:
> 
> > Hi Guys
> >
> > I want to customize the collector output file name and the storing path by
> > the log file name.
> > Such as if the log name is a-b.log
> > I want the storing path is like /a/b/a-b.log
> >
> > Is it possible?
> >
> >
> >
>