You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Jintao Guan <il...@gmail.com> on 2013/06/04 20:06:20 UTC
How to use Esper in Flume
Hello guys,
I want to use Esper in Flume Agents, so that I can monitor all of the
messages and logs and trigger alerts when there are Error logs.
Does it seem like a plugin ? How do I install the custom interceptor ?
Does anyone have any idea ?
RE: How to use Esper in Flume
Posted by Phil Scala <Ph...@globalrelay.net>.
Jintao, take a look here: http://flume.apache.org/FlumeDeveloperGuide.html#sink
It's pretty straight forward (coming from a .net guy :)
1. Create your sink based on the dev guide into a jar
2. Set your config file to use that sink type:
spool-agent.sources = spooler
spool-agent.channels = FileChannel-1
spool-agent.sinks = mysink
spool-agent.sinks.mysink.type = com.domain.flume.sink.esper.EsperSink
spool-agent.sinks.mysink.esperAddress = 10.50.2.131 //can read this value in your configure(Context context) implementation
spool-agent.sinks.mysink.channel = FileChannel-1
3. Add your sink jar to the classpath
Phil
Phil Scala
Software Developer / Architect
Global Relay
phil.scala@globalrelay.net<ma...@globalrelay.net>
866.484.6630 | info@globalrelay.net<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net> | globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f>
From: Jintao Guan [mailto:ilovexiao77@gmail.com]
Sent: Tuesday, June 04, 2013 2:49 PM
To: user@flume.apache.org
Subject: Re: How to use Esper in Flume
Hi Shekhar,
Thank you for your email. The first solution is what I am looking for.
I really want to know the details of the custom sink.
1. The first thing I want to know is that how to install the custom sink ? Compile it then put the ***.jar into $FLOME_HOME/lib ?
2. Is there any material about how to write custom sink ?
Thank you.
On Tue, Jun 4, 2013 at 2:41 PM, Shekhar Sharma <sh...@gmail.com>> wrote:
You have two options :
(1) You can write a custom flume sink for esper..Disadvantage would be you cant change the esper queries at run time
(2) YOu can write esper service (Rest API) and the flume sink will send the java pojo to Esper service..The advantage is that you will have the ability to change the queries at run time..
I have tried both the approaches..let me know if you would like to know more...
The first approach is very simple, just need to extend the sink from flume Abstract sink and override the process method..
In the constructor of the flume you need to initialze the esper engine, as well as esper statements and esper queries..
Make sure your listeners are attached as well when constructing the object..
While you receive the events in the process method, you need to convert the Flume SimpleEvent to POJO and call EsperEngine's sendevent() method...
For the second method you can have your queries,listeners and statements defined in XML file and you parse them when the service starts and have a thread that looks for the change of this xml file..
Hope that helps.
Regards,
Som Shekhar Sharma
+91-8197243810<tel:%2B91-8197243810>
On Tue, Jun 4, 2013 at 11:36 PM, Jintao Guan <il...@gmail.com>> wrote:
Hello guys,
I want to use Esper in Flume Agents, so that I can monitor all of the messages and logs and trigger alerts when there are Error logs.
Does it seem like a plugin ? How do I install the custom interceptor ?
Does anyone have any idea ?
Re: How to use Esper in Flume
Posted by Roshan Naik <ro...@hortonworks.com>.
perhaps you can contribute this back into open source.
On Wed, Jun 5, 2013 at 8:02 AM, Jintao Guan <il...@gmail.com> wrote:
> Thank you guys.
> Now I have written my custom sink of using Esper in Flume.
>
>
> On Tue, Jun 4, 2013 at 2:49 PM, Jintao Guan <il...@gmail.com> wrote:
>
>> Hi Shekhar,
>>
>> Thank you for your email. The first solution is what I am looking for.
>> I really want to know the details of the custom sink.
>>
>> 1. The first thing I want to know is that how to install the custom sink
>> ? Compile it then put the ***.jar into $FLOME_HOME/lib ?
>> 2. Is there any material about how to write custom sink ?
>>
>> Thank you.
>>
>>
>> On Tue, Jun 4, 2013 at 2:41 PM, Shekhar Sharma <sh...@gmail.com>wrote:
>>
>>> You have two options :
>>>
>>> (1) You can write a custom flume sink for esper..Disadvantage would be
>>> you cant change the esper queries at run time
>>>
>>> (2) YOu can write esper service (Rest API) and the flume sink will send
>>> the java pojo to Esper service..The advantage is that you will have the
>>> ability to change the queries at run time..
>>>
>>> I have tried both the approaches..let me know if you would like to know
>>> more...
>>> The first approach is very simple, just need to extend the sink from
>>> flume Abstract sink and override the process method..
>>> In the constructor of the flume you need to initialze the esper engine,
>>> as well as esper statements and esper queries..
>>> Make sure your listeners are attached as well when constructing the
>>> object..
>>> While you receive the events in the process method, you need to convert
>>> the Flume SimpleEvent to POJO and call EsperEngine's sendevent() method...
>>>
>>> For the second method you can have your queries,listeners and statements
>>> defined in XML file and you parse them when the service starts and have a
>>> thread that looks for the change of this xml file..
>>>
>>> Hope that helps.
>>> Regards,
>>> Som Shekhar Sharma
>>> +91-8197243810
>>>
>>>
>>> On Tue, Jun 4, 2013 at 11:36 PM, Jintao Guan <il...@gmail.com>wrote:
>>>
>>>> Hello guys,
>>>>
>>>> I want to use Esper in Flume Agents, so that I can monitor all of the
>>>> messages and logs and trigger alerts when there are Error logs.
>>>>
>>>> Does it seem like a plugin ? How do I install the custom interceptor ?
>>>>
>>>> Does anyone have any idea ?
>>>>
>>>
>>>
>>
>
Re: How to use Esper in Flume
Posted by Jintao Guan <il...@gmail.com>.
Thank you guys.
Now I have written my custom sink of using Esper in Flume.
On Tue, Jun 4, 2013 at 2:49 PM, Jintao Guan <il...@gmail.com> wrote:
> Hi Shekhar,
>
> Thank you for your email. The first solution is what I am looking for.
> I really want to know the details of the custom sink.
>
> 1. The first thing I want to know is that how to install the custom sink ?
> Compile it then put the ***.jar into $FLOME_HOME/lib ?
> 2. Is there any material about how to write custom sink ?
>
> Thank you.
>
>
> On Tue, Jun 4, 2013 at 2:41 PM, Shekhar Sharma <sh...@gmail.com>wrote:
>
>> You have two options :
>>
>> (1) You can write a custom flume sink for esper..Disadvantage would be
>> you cant change the esper queries at run time
>>
>> (2) YOu can write esper service (Rest API) and the flume sink will send
>> the java pojo to Esper service..The advantage is that you will have the
>> ability to change the queries at run time..
>>
>> I have tried both the approaches..let me know if you would like to know
>> more...
>> The first approach is very simple, just need to extend the sink from
>> flume Abstract sink and override the process method..
>> In the constructor of the flume you need to initialze the esper engine,
>> as well as esper statements and esper queries..
>> Make sure your listeners are attached as well when constructing the
>> object..
>> While you receive the events in the process method, you need to convert
>> the Flume SimpleEvent to POJO and call EsperEngine's sendevent() method...
>>
>> For the second method you can have your queries,listeners and statements
>> defined in XML file and you parse them when the service starts and have a
>> thread that looks for the change of this xml file..
>>
>> Hope that helps.
>> Regards,
>> Som Shekhar Sharma
>> +91-8197243810
>>
>>
>> On Tue, Jun 4, 2013 at 11:36 PM, Jintao Guan <il...@gmail.com>wrote:
>>
>>> Hello guys,
>>>
>>> I want to use Esper in Flume Agents, so that I can monitor all of the
>>> messages and logs and trigger alerts when there are Error logs.
>>>
>>> Does it seem like a plugin ? How do I install the custom interceptor ?
>>>
>>> Does anyone have any idea ?
>>>
>>
>>
>
Re: How to use Esper in Flume
Posted by Jintao Guan <il...@gmail.com>.
Hi Shekhar,
Thank you for your email. The first solution is what I am looking for.
I really want to know the details of the custom sink.
1. The first thing I want to know is that how to install the custom sink ?
Compile it then put the ***.jar into $FLOME_HOME/lib ?
2. Is there any material about how to write custom sink ?
Thank you.
On Tue, Jun 4, 2013 at 2:41 PM, Shekhar Sharma <sh...@gmail.com>wrote:
> You have two options :
>
> (1) You can write a custom flume sink for esper..Disadvantage would be you
> cant change the esper queries at run time
>
> (2) YOu can write esper service (Rest API) and the flume sink will send
> the java pojo to Esper service..The advantage is that you will have the
> ability to change the queries at run time..
>
> I have tried both the approaches..let me know if you would like to know
> more...
> The first approach is very simple, just need to extend the sink from flume
> Abstract sink and override the process method..
> In the constructor of the flume you need to initialze the esper engine, as
> well as esper statements and esper queries..
> Make sure your listeners are attached as well when constructing the
> object..
> While you receive the events in the process method, you need to convert
> the Flume SimpleEvent to POJO and call EsperEngine's sendevent() method...
>
> For the second method you can have your queries,listeners and statements
> defined in XML file and you parse them when the service starts and have a
> thread that looks for the change of this xml file..
>
> Hope that helps.
> Regards,
> Som Shekhar Sharma
> +91-8197243810
>
>
> On Tue, Jun 4, 2013 at 11:36 PM, Jintao Guan <il...@gmail.com>wrote:
>
>> Hello guys,
>>
>> I want to use Esper in Flume Agents, so that I can monitor all of the
>> messages and logs and trigger alerts when there are Error logs.
>>
>> Does it seem like a plugin ? How do I install the custom interceptor ?
>>
>> Does anyone have any idea ?
>>
>
>
Re: How to use Esper in Flume
Posted by Shekhar Sharma <sh...@gmail.com>.
You have two options :
(1) You can write a custom flume sink for esper..Disadvantage would be you
cant change the esper queries at run time
(2) YOu can write esper service (Rest API) and the flume sink will send the
java pojo to Esper service..The advantage is that you will have the ability
to change the queries at run time..
I have tried both the approaches..let me know if you would like to know
more...
The first approach is very simple, just need to extend the sink from flume
Abstract sink and override the process method..
In the constructor of the flume you need to initialze the esper engine, as
well as esper statements and esper queries..
Make sure your listeners are attached as well when constructing the object..
While you receive the events in the process method, you need to convert the
Flume SimpleEvent to POJO and call EsperEngine's sendevent() method...
For the second method you can have your queries,listeners and statements
defined in XML file and you parse them when the service starts and have a
thread that looks for the change of this xml file..
Hope that helps.
Regards,
Som Shekhar Sharma
+91-8197243810
On Tue, Jun 4, 2013 at 11:36 PM, Jintao Guan <il...@gmail.com> wrote:
> Hello guys,
>
> I want to use Esper in Flume Agents, so that I can monitor all of the
> messages and logs and trigger alerts when there are Error logs.
>
> Does it seem like a plugin ? How do I install the custom interceptor ?
>
> Does anyone have any idea ?
>