You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Jonathan Hsieh <jo...@cloudera.com> on 2011/07/26 00:41:08 UTC
Re: filter out events
[Please subscribe to new flume-user@incubator.apache.org list, bcc
flume-user@cloudera.org, cc flume-user@incubator.apache.org]
Lior,
You could write a decorator plugin in that only propagates the message that
meet a particular criteria and not send the others.
If you used BE or DFO mode, you could put this at the agent or the
collector.
If you are using this with the e2e mode, this would need to go to the agent.
This is because you would need to make sure that this filter happens
*before* it gets to the agentE2E. If it is after, you will remove messages
that are needed to calculate a proper checksum!
Jon.
On Tue, Jul 12, 2011 at 11:21 PM, Lior H <ha...@gmail.com> wrote:
> I am looking for a way to drop events from flowing down the flume
> stream.
> For example, when tailing a directory of log files from different
> processes, how would one collect errors and warnings only?
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com
Re: filter out events
Posted by NerdyNick <ne...@gmail.com>.
Feature Jira is put in https://issues.apache.org/jira/browse/FLUME-723
On Fri, Jul 29, 2011 at 9:55 AM, Lior Harel <ha...@gmail.com> wrote:
>> Hi Nick,
>> That's definitely sounds like a good idea, a moment after submitting my questions I noticed there are similar questions in the flume users group. So I guess such functionality will be valuable.
>> Lior
> On Jul 27, 2011, at 6:19 PM, NerdyNick wrote:
>
>> Seeing as this idea of filtering events comes up a lot do you think a
>> decorator built into Flume that takes a RegEx and if it matches it
>> doesn't send the event would solve most use-cases out there. If this
>> sounds like a good idea I'll throw a Jira task up to do it as well as
>> make notes about the limitations to make sure they get into the docs
>> about the decorator. Of course we got to wait tell the new Jira is
>> ready.
>>
>> On Mon, Jul 25, 2011 at 4:41 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>>> [Please subscribe to new flume-user@incubator.apache.org list,
>>> bcc flume-user@cloudera.org, cc flume-user@incubator.apache.org]
>>> Lior,
>>> You could write a decorator plugin in that only propagates the message that
>>> meet a particular criteria and not send the others.
>>> If you used BE or DFO mode, you could put this at the agent or the
>>> collector.
>>> If you are using this with the e2e mode, this would need to go to the agent.
>>> This is because you would need to make sure that this filter happens
>>> *before* it gets to the agentE2E. If it is after, you will remove messages
>>> that are needed to calculate a proper checksum!
>>> Jon.
>>>
>>> On Tue, Jul 12, 2011 at 11:21 PM, Lior H <ha...@gmail.com> wrote:
>>>>
>>>> I am looking for a way to drop events from flowing down the flume
>>>> stream.
>>>> For example, when tailing a directory of log files from different
>>>> processes, how would one collect errors and warnings only?
>>>
>>>
>>>
>>> --
>>> // Jonathan Hsieh (shay)
>>> // Software Engineer, Cloudera
>>> // jon@cloudera.com
>>>
>>>
>>
>>
>>
>> --
>> Nick Verbeck - NerdyNick
>> ----------------------------------------------------
>> NerdyNick.com
>> Coloco.ubuntu-rocks.org
>
>
--
Nick Verbeck - NerdyNick
----------------------------------------------------
NerdyNick.com
Coloco.ubuntu-rocks.org
Re: filter out events
Posted by Lior Harel <ha...@gmail.com>.
> Hi Nick,
> That's definitely sounds like a good idea, a moment after submitting my questions I noticed there are similar questions in the flume users group. So I guess such functionality will be valuable.
> Lior
On Jul 27, 2011, at 6:19 PM, NerdyNick wrote:
> Seeing as this idea of filtering events comes up a lot do you think a
> decorator built into Flume that takes a RegEx and if it matches it
> doesn't send the event would solve most use-cases out there. If this
> sounds like a good idea I'll throw a Jira task up to do it as well as
> make notes about the limitations to make sure they get into the docs
> about the decorator. Of course we got to wait tell the new Jira is
> ready.
>
> On Mon, Jul 25, 2011 at 4:41 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>> [Please subscribe to new flume-user@incubator.apache.org list,
>> bcc flume-user@cloudera.org, cc flume-user@incubator.apache.org]
>> Lior,
>> You could write a decorator plugin in that only propagates the message that
>> meet a particular criteria and not send the others.
>> If you used BE or DFO mode, you could put this at the agent or the
>> collector.
>> If you are using this with the e2e mode, this would need to go to the agent.
>> This is because you would need to make sure that this filter happens
>> *before* it gets to the agentE2E. If it is after, you will remove messages
>> that are needed to calculate a proper checksum!
>> Jon.
>>
>> On Tue, Jul 12, 2011 at 11:21 PM, Lior H <ha...@gmail.com> wrote:
>>>
>>> I am looking for a way to drop events from flowing down the flume
>>> stream.
>>> For example, when tailing a directory of log files from different
>>> processes, how would one collect errors and warnings only?
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // Software Engineer, Cloudera
>> // jon@cloudera.com
>>
>>
>
>
>
> --
> Nick Verbeck - NerdyNick
> ----------------------------------------------------
> NerdyNick.com
> Coloco.ubuntu-rocks.org
Re: filter out events
Posted by NerdyNick <ne...@gmail.com>.
Seeing as this idea of filtering events comes up a lot do you think a
decorator built into Flume that takes a RegEx and if it matches it
doesn't send the event would solve most use-cases out there. If this
sounds like a good idea I'll throw a Jira task up to do it as well as
make notes about the limitations to make sure they get into the docs
about the decorator. Of course we got to wait tell the new Jira is
ready.
On Mon, Jul 25, 2011 at 4:41 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> [Please subscribe to new flume-user@incubator.apache.org list,
> bcc flume-user@cloudera.org, cc flume-user@incubator.apache.org]
> Lior,
> You could write a decorator plugin in that only propagates the message that
> meet a particular criteria and not send the others.
> If you used BE or DFO mode, you could put this at the agent or the
> collector.
> If you are using this with the e2e mode, this would need to go to the agent.
> This is because you would need to make sure that this filter happens
> *before* it gets to the agentE2E. If it is after, you will remove messages
> that are needed to calculate a proper checksum!
> Jon.
>
> On Tue, Jul 12, 2011 at 11:21 PM, Lior H <ha...@gmail.com> wrote:
>>
>> I am looking for a way to drop events from flowing down the flume
>> stream.
>> For example, when tailing a directory of log files from different
>> processes, how would one collect errors and warnings only?
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>
--
Nick Verbeck - NerdyNick
----------------------------------------------------
NerdyNick.com
Coloco.ubuntu-rocks.org