You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Ma...@itcinfotech.com on 2014/09/26 14:24:34 UTC
Programmatically configuring new source/sink into Flume agent
Hi All,
I have a use-case where in I get source and sink details from user via UI page, these config details I want to feed to flum so that programmatically it starts processing newly added request for source/sink. When I was looking at developer guide https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift there is mention of using RPC client to programmatically get handle to flume instance, now I wanted to know how can I programmatically feed these user details as and when they arrive ?
Example - one definition may look like copy files from NFs mount to HDFS and one more might look like copy zipped files placed in other directory on local system to HDFS sink etc.
Thanks,
Manohar.
Please consider the environment before printing this e-mail
Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
Re: Programmatically configuring new source/sink into Flume agent
Posted by Ashish <pa...@gmail.com>.
AFAIK, you cannot add anything to a running agent. You directly
programatically create configuration. Have a look at the test cases, they
all create configurations programatically. Can also look at Embedded Agent.
RPC is a different thing, meant to send data to Flume Source via RPC. Can
you point to the section where it says you can get handle to Flume Agent?
HTH !
On Mon, Sep 29, 2014 at 10:28 AM, <Ma...@itcinfotech.com> wrote:
> Yes Blade,
>
>
>
> My requirement is either programmatically re-configure config file or add
> new source/sink definitions to the already running agent.
>
>
>
> Will RPC client serves this purpose?Any examples will help me explore more
> on it.
>
>
>
> Thanks,
>
> Manohar.
>
>
>
> *From:* Blade Liu [mailto:hafzcdcn@gmail.com]
> *Sent:* Sunday, September 28, 2014 2:05 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Programmatically configuring new source/sink into Flume
> agent
>
>
>
> I guess you want to reconfig and run flume agents on-the-fly. Using RPC is
> intended to restart agents or handle new config?
>
>
>
>
>
> Thanks,
>
> Blade
>
>
>
> 2014-09-27 7:58 GMT+08:00 terreyshih <te...@gmail.com>:
>
> HI, Manohar:
>
>
>
> I don’t understand your examples. copying/moving file from one directory
> to another is not what source / sinks do.
>
>
>
> thx
>
>
>
> On Sep 26, 2014, at 5:24 AM, Manohar.CS@itcinfotech.com wrote:
>
>
>
> Hi All,
>
>
>
> I have a use-case where in I get source and sink details from user via UI
> page, these config details I want to feed to flum so that programmatically
> it starts processing newly added request for source/sink. When I was
> looking at developer guide
> https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift there
> is mention of using RPC client to programmatically get handle to flume
> instance, now I wanted to know how can I programmatically feed these user
> details as and when they arrive ?
>
>
>
> Example – one definition may look like copy files from NFs mount to HDFS
> and one more might look like copy zipped files placed in other directory on
> local system to HDFS sink etc.
>
>
>
> Thanks,
>
> Manohar.
>
>
>
>
>
>
>
>
>
> Please consider the environment before printing this e-mail
>
>
> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Programmatically configuring new source/sink into Flume agent
Posted by Satish Abburi <sa...@gmail.com>.
Did anyone build UI for configuring and deploying flume configuration.. We
are trying to build one, want to check if there is something already exists
in the community, that can be leveraged.
- Deploy 1..n flume agents in a enterprise
- add sources/sinks/channels/interceptors
- connect the agents
Thanks,
Satish
On Sun, Sep 28, 2014 at 10:21 PM, terreyshih <te...@gmail.com> wrote:
> I think if the requirement is to reconfigure the config file by adding,
> removing or provisioning and existing agent, you don’t really have to make
> it too complicated. Just modify the content of your .conf file
> programmatically via UI.
>
> The flume agent reloads the conf file periodically, like every 15 seconds
> or something so your changes will be propagated.
>
> This is the simplest way without any RPC.
>
> thanks,
> -Terrey
>
> On Sep 28, 2014, at 9:58 PM, Manohar.CS@itcinfotech.com wrote:
>
> Yes Blade,
>
> My requirement is either programmatically re-configure config file or add
> new source/sink definitions to the already running agent.
>
> Will RPC client serves this purpose?Any examples will help me explore more
> on it.
>
> Thanks,
> Manohar.
>
> *From:* Blade Liu [mailto:hafzcdcn@gmail.com <ha...@gmail.com>]
> *Sent:* Sunday, September 28, 2014 2:05 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Programmatically configuring new source/sink into Flume
> agent
>
> I guess you want to reconfig and run flume agents on-the-fly. Using RPC is
> intended to restart agents or handle new config?
>
>
> Thanks,
> Blade
>
> 2014-09-27 7:58 GMT+08:00 terreyshih <te...@gmail.com>:
>
> HI, Manohar:
>
> I don’t understand your examples. copying/moving file from one directory
> to another is not what source / sinks do.
>
> thx
>
> On Sep 26, 2014, at 5:24 AM, Manohar.CS@itcinfotech.com wrote:
>
>
> Hi All,
>
> I have a use-case where in I get source and sink details from user via UI
> page, these config details I want to feed to flum so that programmatically
> it starts processing newly added request for source/sink. When I was
> looking at developer guide
> https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift there
> is mention of using RPC client to programmatically get handle to flume
> instance, now I wanted to know how can I programmatically feed these user
> details as and when they arrive ?
>
> Example – one definition may look like copy files from NFs mount to HDFS
> and one more might look like copy zipped files placed in other directory on
> local system to HDFS sink etc.
>
> Thanks,
> Manohar.
>
>
>
>
>
>
>
>
> Please consider the environment before printing this e-mail
>
>
> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>
>
>
Re: Programmatically configuring new source/sink into Flume agent
Posted by terreyshih <te...@gmail.com>.
I think if the requirement is to reconfigure the config file by adding, removing or provisioning and existing agent, you don’t really have to make it too complicated. Just modify the content of your .conf file programmatically via UI.
The flume agent reloads the conf file periodically, like every 15 seconds or something so your changes will be propagated.
This is the simplest way without any RPC.
thanks,
-Terrey
On Sep 28, 2014, at 9:58 PM, Manohar.CS@itcinfotech.com wrote:
> Yes Blade,
>
> My requirement is either programmatically re-configure config file or add new source/sink definitions to the already running agent.
>
> Will RPC client serves this purpose?Any examples will help me explore more on it.
>
> Thanks,
> Manohar.
>
> From: Blade Liu [mailto:hafzcdcn@gmail.com]
> Sent: Sunday, September 28, 2014 2:05 PM
> To: user@flume.apache.org
> Subject: Re: Programmatically configuring new source/sink into Flume agent
>
> I guess you want to reconfig and run flume agents on-the-fly. Using RPC is intended to restart agents or handle new config?
>
>
> Thanks,
> Blade
>
> 2014-09-27 7:58 GMT+08:00 terreyshih <te...@gmail.com>:
> HI, Manohar:
>
> I don’t understand your examples. copying/moving file from one directory to another is not what source / sinks do.
>
> thx
>
> On Sep 26, 2014, at 5:24 AM, Manohar.CS@itcinfotech.com wrote:
>
>
> Hi All,
>
> I have a use-case where in I get source and sink details from user via UI page, these config details I want to feed to flum so that programmatically it starts processing newly added request for source/sink. When I was looking at developer guide https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift there is mention of using RPC client to programmatically get handle to flume instance, now I wanted to know how can I programmatically feed these user details as and when they arrive ?
>
> Example – one definition may look like copy files from NFs mount to HDFS and one more might look like copy zipped files placed in other directory on local system to HDFS sink etc.
>
> Thanks,
> Manohar.
>
>
>
>
>
>
> Please consider the environment before printing this e-mail
>
> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
RE: Programmatically configuring new source/sink into Flume agent
Posted by Ma...@itcinfotech.com.
Yes Blade,
My requirement is either programmatically re-configure config file or add new source/sink definitions to the already running agent.
Will RPC client serves this purpose?Any examples will help me explore more on it.
Thanks,
Manohar.
From: Blade Liu [mailto:hafzcdcn@gmail.com]
Sent: Sunday, September 28, 2014 2:05 PM
To: user@flume.apache.org
Subject: Re: Programmatically configuring new source/sink into Flume agent
I guess you want to reconfig and run flume agents on-the-fly. Using RPC is intended to restart agents or handle new config?
Thanks,
Blade
2014-09-27 7:58 GMT+08:00 terreyshih <te...@gmail.com>>:
HI, Manohar:
I don’t understand your examples. copying/moving file from one directory to another is not what source / sinks do.
thx
On Sep 26, 2014, at 5:24 AM, Manohar.CS@itcinfotech.com<ma...@itcinfotech.com> wrote:
Hi All,
I have a use-case where in I get source and sink details from user via UI page, these config details I want to feed to flum so that programmatically it starts processing newly added request for source/sink. When I was looking at developer guide https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift there is mention of using RPC client to programmatically get handle to flume instance, now I wanted to know how can I programmatically feed these user details as and when they arrive ?
Example – one definition may look like copy files from NFs mount to HDFS and one more might look like copy zipped files placed in other directory on local system to HDFS sink etc.
Thanks,
Manohar.
Please consider the environment before printing this e-mail
Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
Re: Programmatically configuring new source/sink into Flume agent
Posted by Blade Liu <ha...@gmail.com>.
I guess you want to reconfig and run flume agents on-the-fly. Using RPC is
intended to restart agents or handle new config?
Thanks,
Blade
2014-09-27 7:58 GMT+08:00 terreyshih <te...@gmail.com>:
> HI, Manohar:
>
> I don’t understand your examples. copying/moving file from one directory
> to another is not what source / sinks do.
>
> thx
>
> On Sep 26, 2014, at 5:24 AM, Manohar.CS@itcinfotech.com wrote:
>
> Hi All,
>
> I have a use-case where in I get source and sink details from user via UI
> page, these config details I want to feed to flum so that programmatically
> it starts processing newly added request for source/sink. When I was
> looking at developer guide
> https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift
> there is mention of using RPC client to programmatically get handle to
> flume instance, now I wanted to know how can I programmatically feed these
> user details as and when they arrive ?
>
> Example – one definition may look like copy files from NFs mount to HDFS
> and one more might look like copy zipped files placed in other directory on
> local system to HDFS sink etc.
>
> Thanks,
> Manohar.
>
>
>
Re: Programmatically configuring new source/sink into Flume agent
Posted by terreyshih <te...@gmail.com>.
HI, Manohar:
I don’t understand your examples. copying/moving file from one directory to another is not what source / sinks do.
thx
On Sep 26, 2014, at 5:24 AM, Manohar.CS@itcinfotech.com wrote:
> Hi All,
>
> I have a use-case where in I get source and sink details from user via UI page, these config details I want to feed to flum so that programmatically it starts processing newly added request for source/sink. When I was looking at developer guide https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift there is mention of using RPC client to programmatically get handle to flume instance, now I wanted to know how can I programmatically feed these user details as and when they arrive ?
>
> Example – one definition may look like copy files from NFs mount to HDFS and one more might look like copy zipped files placed in other directory on local system to HDFS sink etc.
>
> Thanks,
> Manohar.
>