You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Ma...@itcinfotech.com on 2014/09/16 14:05:07 UTC
Configuring flume agent dynamically from java code
Hi Flume developers,
I have an use-case where in I need to configure the flume agent dynamically/programmatically via Java and start the flume agent. Note that these agent configuration details will be fed in by User in UI which Java code will translate to flume agent properties. Please let me know how I can go about this use case.
Thanks,
Manohar.
Please consider the environment before printing this e-mail
Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
Re: Configuring flume agent dynamically from java code
Posted by Blade Liu <ha...@gmail.com>.
I guess you can assign customized parameters(provided by UI) using Java
code and directly write file to flume.conf. It would be nice if someone
develops a general framework...
2014-09-16 20:57 GMT+08:00 Ahmed Vila <av...@devlogic.eu>:
> Hi Manohar,
>
> I must have miss read your question. You actually want to make UI
> configurator (not a full support) for Flume running as a service, correct ?
> I can tell you from my experience that flume monitors it's configuration
> file and reloads once there is a change within a minute or so, without
> restarting it's JVM.
>
>
> On Tue, Sep 16, 2014 at 2:36 PM, <Ma...@itcinfotech.com> wrote:
>
>> HI Ahmed,
>>
>>
>>
>> I already went through embedded flume client, as doc suggests its
>> lightweight implementation of Flume functionalities.In my use case I am
>> planning to make use of “Sink.type= spoolDir”,
>>
>> I doubt I will support full fledge functionalities.
>>
>>
>>
>> Thanks,
>>
>> Manohar.
>>
>>
>>
>> *From:* Ahmed Vila [mailto:avila@devlogic.eu]
>> *Sent:* Tuesday, September 16, 2014 5:58 PM
>> *To:* user@flume.apache.org
>> *Subject:* Re: Configuring flume agent dynamically from java code
>>
>>
>>
>> Hi Manohar,
>>
>>
>>
>> Guess you're looking for an embed Flume client.
>>
>> This might give you rough idea:
>>
>>
>> http://www.javacodegeeks.com/2014/05/flume-gathering-customer-product-search-clicks-data-using-apache-flume.html
>>
>>
>>
>>
>>
>> On Tue, Sep 16, 2014 at 2:05 PM, <Ma...@itcinfotech.com> wrote:
>>
>>
>>
>>
>>
>> Hi Flume developers,
>>
>>
>>
>> I have an use-case where in I need to configure the flume agent
>> dynamically/programmatically via Java and start the flume agent. Note that
>> these agent configuration details will be fed in by User in UI which Java
>> code will translate to flume agent properties. Please let me know how I can
>> go about this use case.
>>
>>
>>
>> Thanks,
>>
>> Manohar.
>>
>>
>>
>>
>> Please consider the environment before printing this e-mail
>>
>>
>> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>>
>>
>>
>>
>>
>> --
>>
>> Best regards,
>>
>> Ahmed Vila
>>
>>
>> ---------------------------------------------------------------------
>> This e-mail and any attachment is for authorised use by the intended
>> recipient(s) only. This email contains confidential information. It should
>> not be copied, disclosed to, retained or used by, any party other than the
>> intended recipient. Any unauthorised distribution, dissemination or copying
>> of this E-mail or its attachments, and/or any use of any information
>> contained in them, is strictly prohibited and may be illegal. If you are
>> not an intended recipient then please promptly delete this e-mail and any
>> attachment and all copies and inform the sender directly via email. Any
>> emails that you send to us may be monitored by systems or persons other
>> than the named communicant for the purposes of ascertaining whether the
>> communication complies with the law and company policies.
>>
>>
>>
>> Please consider the environment before printing this e-mail
>>
>>
>> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>>
>
>
>
> --
>
> Best regards,
> Ahmed Vila
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
Re: Configuring flume agent dynamically from java code
Posted by Ahmed Vila <av...@devlogic.eu>.
Hi Manohar,
I must have miss read your question. You actually want to make UI
configurator (not a full support) for Flume running as a service, correct ?
I can tell you from my experience that flume monitors it's configuration
file and reloads once there is a change within a minute or so, without
restarting it's JVM.
On Tue, Sep 16, 2014 at 2:36 PM, <Ma...@itcinfotech.com> wrote:
> HI Ahmed,
>
>
>
> I already went through embedded flume client, as doc suggests its
> lightweight implementation of Flume functionalities.In my use case I am
> planning to make use of “Sink.type= spoolDir”,
>
> I doubt I will support full fledge functionalities.
>
>
>
> Thanks,
>
> Manohar.
>
>
>
> *From:* Ahmed Vila [mailto:avila@devlogic.eu]
> *Sent:* Tuesday, September 16, 2014 5:58 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Configuring flume agent dynamically from java code
>
>
>
> Hi Manohar,
>
>
>
> Guess you're looking for an embed Flume client.
>
> This might give you rough idea:
>
>
> http://www.javacodegeeks.com/2014/05/flume-gathering-customer-product-search-clicks-data-using-apache-flume.html
>
>
>
>
>
> On Tue, Sep 16, 2014 at 2:05 PM, <Ma...@itcinfotech.com> wrote:
>
>
>
>
>
> Hi Flume developers,
>
>
>
> I have an use-case where in I need to configure the flume agent
> dynamically/programmatically via Java and start the flume agent. Note that
> these agent configuration details will be fed in by User in UI which Java
> code will translate to flume agent properties. Please let me know how I can
> go about this use case.
>
>
>
> Thanks,
>
> Manohar.
>
>
>
>
> Please consider the environment before printing this e-mail
>
>
> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>
>
>
>
>
> --
>
> Best regards,
>
> Ahmed Vila
>
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
>
>
> Please consider the environment before printing this e-mail
>
>
> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>
--
Best regards,
Ahmed Vila
--
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.
RE: Configuring flume agent dynamically from java code
Posted by Ma...@itcinfotech.com.
HI Ahmed,
I already went through embedded flume client, as doc suggests its lightweight implementation of Flume functionalities.In my use case I am planning to make use of “Sink.type= spoolDir”,
I doubt I will support full fledge functionalities.
Thanks,
Manohar.
From: Ahmed Vila [mailto:avila@devlogic.eu]
Sent: Tuesday, September 16, 2014 5:58 PM
To: user@flume.apache.org
Subject: Re: Configuring flume agent dynamically from java code
Hi Manohar,
Guess you're looking for an embed Flume client.
This might give you rough idea:
http://www.javacodegeeks.com/2014/05/flume-gathering-customer-product-search-clicks-data-using-apache-flume.html
On Tue, Sep 16, 2014 at 2:05 PM, <Ma...@itcinfotech.com>> wrote:
Hi Flume developers,
I have an use-case where in I need to configure the flume agent dynamically/programmatically via Java and start the flume agent. Note that these agent configuration details will be fed in by User in UI which Java code will translate to flume agent properties. Please let me know how I can go about this use case.
Thanks,
Manohar.
Please consider the environment before printing this e-mail
Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
--
Best regards,
Ahmed Vila
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipient(s) only. This email contains confidential information. It should not be copied, disclosed to, retained or used by, any party other than the intended recipient. Any unauthorised distribution, dissemination or copying of this E-mail or its attachments, and/or any use of any information contained in them, is strictly prohibited and may be illegal. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender directly via email. Any emails that you send to us may be monitored by systems or persons other than the named communicant for the purposes of ascertaining whether the communication complies with the law and company policies.
Please consider the environment before printing this e-mail
Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
Re: Configuring flume agent dynamically from java code
Posted by Ahmed Vila <av...@devlogic.eu>.
Hi Manohar,
Guess you're looking for an embed Flume client.
This might give you rough idea:
http://www.javacodegeeks.com/2014/05/flume-gathering-customer-product-search-clicks-data-using-apache-flume.html
On Tue, Sep 16, 2014 at 2:05 PM, <Ma...@itcinfotech.com> wrote:
>
>
>
>
> Hi Flume developers,
>
>
>
> I have an use-case where in I need to configure the flume agent
> dynamically/programmatically via Java and start the flume agent. Note that
> these agent configuration details will be fed in by User in UI which Java
> code will translate to flume agent properties. Please let me know how I can
> go about this use case.
>
>
>
> Thanks,
>
> Manohar.
>
>
>
> Please consider the environment before printing this e-mail
>
>
> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>
--
Best regards,
Ahmed Vila
--
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.