You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Srinivasan Subramanian <ss...@hotmail.com> on 2011/11/29 15:10:33 UTC

Output bucketing based on custom data / logic

Just got flume installed on Centos and the basic setup is working great.  I have got a Master, collector and a few agents.  
One additional requirement i have is to do with Output Bucketing.  Based on data that is present in the log message that is being sent to flume, the data has to be slotted into different buckets.  For eg assume that the log files are to be separated into different folders based on a customer name that is present in the log message, how would one go about doing this? 
Lets say the log message sent to Flume is "ABC : Some message".  The collector needs to put this into a folder like /var/log/ABC.  How can this be achieved?  
Alternatively,  Can I use metadata?  As far as i could make out meta data is static and not dynamic as i would need it to be.  
Thanks for all the help.
Regards
Srini

Re: Output bucketing based on custom data / logic

Posted by Shuang <sh...@open42.com>.

You can use the regex decorator to parse out "ABC" and put it in a meta
data field for example named "customer", then in your collectorSink,  use
%{customer} in the destination path.

Shuang

On Tue, Nov 29, 2011 at 6:10 AM, Srinivasan Subramanian <
ssrini_vasan@hotmail.com> wrote:

>  Just got flume installed on Centos and the basic setup is working great.
>  I have got a Master, collector and a few agents.
>
> One additional requirement i have is to do with Output Bucketing.  Based
> on data that is present in the log message that is being sent to flume, the
> data has to be slotted into different buckets.  For eg assume that the log
> files are to be separated into different folders based on a customer name
> that is present in the log message, how would one go about doing this?
>
> Lets say the log message sent to Flume is "ABC : Some message".  The
> collector needs to put this into a folder like /var/log/ABC.  How can this
> be achieved?
>
> Alternatively,  Can I use metadata?  As far as i could make out meta data
> is static and not dynamic as i would need it to be.
>
> Thanks for all the help.
>
> Regards
> Srini
>