You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by 纪君祥 <zh...@alipay.com> on 2011/09/16 07:51:14 UTC
RE: How to avoid restart master when adding one customer own plugin?
Hi all,
Could anyone tell me how to avoid restart master when adding one plugin? Any help are appreciated.
I found that when I implement my own plugin, if I want to use it, I need restart the master node, otherwise, it will report configuration error.
Can we avoid it?
Thanks a lot
Longda
________________________________
This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you.
本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
Re: How to avoid restart master when adding one customer own plugin?
Posted by Mingjie Lai <mj...@gmail.com>.
> But what I worry is restarting master will refresh all configuration,
it will bring trouble for tail/text, or other source/sink.
Configurations can be saved to zookeeper so they won't be lost with a
master restart. Check it out:
<property>
<name>flume.master.store</name>
<value>zookeeper</value>
<description>How the Flume Master stores node configurations. Must
be either 'zookeeper' or 'memory'.</description>
</property>
My config:
<property>
<name>flume.master.store</name>
<value>zookeeper</value>
</property>
<property>
<name>flume.master.zk.use.external</name>
<value>true</value>
</property>
<property>
<name>flume.master.zk.servers</name>
<value>ip-10-160-218-3.us-west-1.compute.internal:2181</value>
</property>
-mingjie
On 09/18/2011 07:09 PM, 纪君祥 wrote:
> Thanks, Mingjie
>
> I think adding one script decorator can works well.
>
> But what I worry is restarting master will refresh all configuration, it will bring trouble for tail/text, or other source/sink.
>
> I have done one test with 'tail' source, if restart master, it will leading to resending old data in the file and missing some data.
>
> Longda
>
> -----Original Message-----
> From: Mingjie Lai [mailto:mjlai09@gmail.com]
> Sent: Saturday, September 17, 2011 3:20 AM
> To: flume-user@incubator.apache.org
> Subject: Re: How to avoid restart master when adding one customer own plugin?
>
> Longda.
>
> AFAIK, Flume doesn't have a way to load a plugin on the fly. The plugin
> class names are in the config file, and shared among all flume nodes +
> master. Also the class files need to be loaded when flume starts up.
> When you have a new plugin, you have to 1) reconfigure flume at all
> nodes, 2) copy the new class file to flume classpath, 3) restart all nodes.
>
> Assume there is a way for flume to dynamically config/load plugin class
> names, we still need a way to ship the class file to each node which can
> not be handled by flume easily.
>
> Personally I don't think it's a strong operational requirement.
>
> Or you can have a script decorator plugin to avoid flume restart:
>
> http://blog.infochimps.com/2011/03/02/simple-flume-decorators-with-jruby/
>
> -mingjie
>
> On 09/15/2011 10:51 PM, 纪君祥 wrote:
>> Hi all,
>>
>> Could anyone tell me how to avoid restart master when adding one plugin?
>> Any help are appreciated.
>>
>> I found that when I implement my own plugin, if I want to use it, I need
>> restart the master node, otherwise, it will report configuration error.
>>
>> Can we avoid it?
>>
>> Thanks a lot
>>
>> Longda
>>
>>
>> ------------------------------------------------------------------------
>>
>> This email (including any attachments) is confidential and may be
>> legally privileged. If you received this email in error, please delete
>> it immediately and do not copy it or use it for any purpose or disclose
>> its contents to any other person. Thank you.
>>
>> 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，
>> 请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮
>> 件之内容。谢谢。
>
> ________________________________
>
> This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you.
>
> 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
RE: How to avoid restart master when adding one customer own plugin?
Posted by 纪君祥 <zh...@alipay.com>.
Thanks, Mingjie
I think adding one script decorator can works well.
But what I worry is restarting master will refresh all configuration, it will bring trouble for tail/text, or other source/sink.
I have done one test with 'tail' source, if restart master, it will leading to resending old data in the file and missing some data.
Longda
-----Original Message-----
From: Mingjie Lai [mailto:mjlai09@gmail.com]
Sent: Saturday, September 17, 2011 3:20 AM
To: flume-user@incubator.apache.org
Subject: Re: How to avoid restart master when adding one customer own plugin?
Longda.
AFAIK, Flume doesn't have a way to load a plugin on the fly. The plugin
class names are in the config file, and shared among all flume nodes +
master. Also the class files need to be loaded when flume starts up.
When you have a new plugin, you have to 1) reconfigure flume at all
nodes, 2) copy the new class file to flume classpath, 3) restart all nodes.
Assume there is a way for flume to dynamically config/load plugin class
names, we still need a way to ship the class file to each node which can
not be handled by flume easily.
Personally I don't think it's a strong operational requirement.
Or you can have a script decorator plugin to avoid flume restart:
http://blog.infochimps.com/2011/03/02/simple-flume-decorators-with-jruby/
-mingjie
On 09/15/2011 10:51 PM, 纪君祥 wrote:
> Hi all,
>
> Could anyone tell me how to avoid restart master when adding one plugin?
> Any help are appreciated.
>
> I found that when I implement my own plugin, if I want to use it, I need
> restart the master node, otherwise, it will report configuration error.
>
> Can we avoid it?
>
> Thanks a lot
>
> Longda
>
>
> ------------------------------------------------------------------------
>
> This email (including any attachments) is confidential and may be
> legally privileged. If you received this email in error, please delete
> it immediately and do not copy it or use it for any purpose or disclose
> its contents to any other person. Thank you.
>
> 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，
> 请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮
> 件之内容。谢谢。
________________________________
This email (including any attachments) is confidential and may be legally privileged. If you received this email in error, please delete it immediately and do not copy it or use it for any purpose or disclose its contents to any other person. Thank you.
本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
Re: How to avoid restart master when adding one customer own plugin?
Posted by Mingjie Lai <mj...@gmail.com>.
Longda.
AFAIK, Flume doesn't have a way to load a plugin on the fly. The plugin
class names are in the config file, and shared among all flume nodes +
master. Also the class files need to be loaded when flume starts up.
When you have a new plugin, you have to 1) reconfigure flume at all
nodes, 2) copy the new class file to flume classpath, 3) restart all nodes.
Assume there is a way for flume to dynamically config/load plugin class
names, we still need a way to ship the class file to each node which can
not be handled by flume easily.
Personally I don't think it's a strong operational requirement.
Or you can have a script decorator plugin to avoid flume restart:
http://blog.infochimps.com/2011/03/02/simple-flume-decorators-with-jruby/
-mingjie
On 09/15/2011 10:51 PM, 纪君祥 wrote:
> Hi all,
>
> Could anyone tell me how to avoid restart master when adding one plugin?
> Any help are appreciated.
>
> I found that when I implement my own plugin, if I want to use it, I need
> restart the master node, otherwise, it will report configuration error.
>
> Can we avoid it?
>
> Thanks a lot
>
> Longda
>
>
> ------------------------------------------------------------------------
>
> This email (including any attachments) is confidential and may be
> legally privileged. If you received this email in error, please delete
> it immediately and do not copy it or use it for any purpose or disclose
> its contents to any other person. Thank you.
>
> 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，
> 请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮
> 件之内容。谢谢。