You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Bertrand Dechoux <de...@gmail.com> on 2013/03/29 14:53:37 UTC
Short guide to custom configuration provider?
Hi,
Flume NG has a pluggable configuration system called the configuration
provider. I cloned the source repository and I indeed found it but I am a
bit lost. Is there some kind of short guide explaining how to write a
custom configuration provider?
I would like to start with something simple : using a YAML configuration
file instead of a properties file.
It should be almost trivial to change the following configuration
*agent1.channels.ch1.type = memory
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 0.0.0.0
agent1.sources.avro-source1.port = 41414
agent1.sinks.log-sink1.channel = ch1
agent1.sinks.log-sink1.type = logger
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = log-sink1*
into
agent1:
channels:
ch1:
type: memory
sources:
avroSource1:
channels: ch1
type: avro
bind: 0.0.0.0
port: 41414
sinks:
logSink1:
channel: ch1
type: logger
At least, I know how to convert the latter into the former but I am not
sure how this transformation can be included into flume-ng using a
configuration provider.
I have not been able to find much information about the customisation of
the configuration provider. Did I miss any documentation?
Bertrand
Re: Short guide to custom configuration provider?
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Bertrand,
https://issues.apache.org/jira/browse/FLUME-1630 changed a lot of the config system, and made it much easier to write a config provider. Once you write a provider you can update the application class to use reflection to instantiate the provider. I dont remember how the patch on FLUME-1491 looks like, but if it is possible to integrate it into this, it might make sense.
Thanks
Hari
--
Hari Shreedharan
On Tuesday, April 2, 2013 at 12:23 PM, Bertrand Dechoux wrote:
> I did a short proof of concept for the YAML part here
> https://github.com/BertrandDechoux/flume-ng-yaml
>
> Does your reaction mean that FLUME-1491 is a 'zombie'? Anyway, I will look at what was done in that issue and see what can be done for a clean selection of a custom configuration provider.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 8:06 PM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > I actually do like the idea of pluggable config provider. Like you mentioned, it should be possible to hack the Application class, and use a CLI arg to select the config provider. If no CLI arg is provided, just fall back to the current mechanism. If you submit a patch, I'd be glad to review it :-)
> >
> >
> > Thanks
> > Hari
> >
> > --
> > Hari Shreedharan
> >
> >
> > On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
> >
> > > It seems quite easier than dealing with commons configuration.
> > >
> > > I might take a shot at it during the week-end, if I find the time.
> > >
> > > About the customisation of the Application class, that's something which is already mostly done by
> > > https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration from Zookeeper watcher
> > >
> > > It might be a good idea to split this improvement into :
> > > 1) Selection of an alternative configuration provider
> > > 2) A zookeeper based configuration provider
> > >
> > > If point 1 is done, anyone could use her/his custom configuration provider by dropping the implementation as a third party jar and then selecting it with the help of the command line.
> > >
> > > Bertrand
> > >
> > >
> > > On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <dechouxb@gmail.com (mailto:dechouxb@gmail.com)> wrote:
> > > > Thanks for the hints!
> > > >
> > > > The part that you quoted is in fact from the flume wiki and I forgot to quote it myself. But the page is quite old so I wasn't sure about the state of the feature. I thought maybe commons-configuration was used but wasn't be able to confirm it clearly (or more exactly I lost myself in this dependency, which might be used by one of my current projects somehow).
> > > >
> > > > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
> > > >
> > > > > Configuration Provider
> > > > >
> > > > > Flume NG has a pluggable configuration system called the configuration
> > > > > provider. By default, Flume NG ships with a Java property file based
> > > > > configuration system that is both simple and easy to generate
> > > > > programmatically. Flume OG has a centralized configuration system with
> > > > > a master and ZooKeeper for coordination and we recognize this is very
> > > > > appealing to some users where as others see it as overhead they simply
> > > > > don't want. We opted to make this a pluggable extension point and ship
> > > > > a basic implementation that would let many users get started quickly
> > > > > and easily. There's almost certainly enough desire for a similar
> > > > > implementation to that of Flume OG, but it isn't yet implemented.
> > > > > Users may also implement arbitrary plugins to integrate with any type
> > > > > of configuration system (JSON files, a shared RDBMS, a central system
> > > > > with ZooKeeper, and so forth). We see this as something more
> > > > > interesting to system integrators.
> > > >
> > > > I will look at the classes you pointed for my own education and I will see what I can do.
> > > >
> > > > Thanks a lot.
> > > >
> > > > Bertrand
> > > >
> > > >
> > > > On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)> wrote:
> > > > > Hi,
> > > > >
> > > > > "Flume NG has a pluggable configuration system called the configuration provider."
> > > > >
> > > > > There is an object hierarchy, but I don't consider it pluggable as there is no place to "plugin" your own implementation. I consider these things "private" to Flume. Meaning that there is no guaranteed API stability between releases. With that caveat, you'd want to implement the following abstract class:
> > > > >
> > > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
> > > > >
> > > > > e.g https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
> > > > >
> > > > > and then'd you have to customize the Application class to "plugin" your own implementation:
> > > > >
> > > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
> > > > >
> > > > > Brock
> > > > >
> > > > >
> > > > > On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <dechouxb@gmail.com (mailto:dechouxb@gmail.com)> wrote:
> > > > > > Hi,
> > > > > >
> > > > > > Flume NG has a pluggable configuration system called the configuration provider. I cloned the source repository and I indeed found it but I am a bit lost. Is there some kind of short guide explaining how to write a custom configuration provider?
> > > > > >
> > > > > > I would like to start with something simple : using a YAML configuration file instead of a properties file.
> > > > > > It should be almost trivial to change the following configuration
> > > > > >
> > > > > > agent1.channels.ch1.type = memory
> > > > > >
> > > > > > agent1.sources.avro-source1.channels = ch1
> > > > > > agent1.sources.avro-source1.type = avro
> > > > > > agent1.sources.avro-source1.bind = 0.0.0.0
> > > > > > agent1.sources.avro-source1.port = 41414
> > > > > >
> > > > > > agent1.sinks.log-sink1.channel = ch1
> > > > > > agent1.sinks.log-sink1.type = logger
> > > > > >
> > > > > > agent1.channels = ch1
> > > > > > agent1.sources = avro-source1
> > > > > > agent1.sinks = log-sink1
> > > > > >
> > > > > > into
> > > > > >
> > > > > > agent1:
> > > > > >
> > > > > > channels:
> > > > > > ch1:
> > > > > > type: memory
> > > > > >
> > > > > > sources:
> > > > > > avroSource1:
> > > > > > channels: ch1
> > > > > > type: avro
> > > > > > bind: 0.0.0.0
> > > > > > port: 41414
> > > > > >
> > > > > > sinks:
> > > > > > logSink1:
> > > > > > channel: ch1
> > > > > > type: logger
> > > > > >
> > > > > >
> > > > > > At least, I know how to convert the latter into the former but I am not sure how this transformation can be included into flume-ng using a configuration provider.
> > > > > > I have not been able to find much information about the customisation of the configuration provider. Did I miss any documentation?
> > > > > >
> > > > > >
> > > > > > Bertrand
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
> > >
> > >
> > >
> > > --
> > > Bertrand Dechoux
> >
>
>
>
> --
> Bertrand Dechoux
Re: Short guide to custom configuration provider?
Posted by Bertrand Dechoux <de...@gmail.com>.
I did a short proof of concept for the YAML part here
https://github.com/BertrandDechoux/flume-ng-yaml
Does your reaction mean that FLUME-1491 is a 'zombie'? Anyway, I will look
at what was done in that issue and see what can be done for a clean
selection of a custom configuration provider.
Bertrand
On Fri, Mar 29, 2013 at 8:06 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:
> I actually do like the idea of pluggable config provider. Like you
> mentioned, it should be possible to hack the Application class, and use a
> CLI arg to select the config provider. If no CLI arg is provided, just fall
> back to the current mechanism. If you submit a patch, I'd be glad to review
> it :-)
>
>
> Thanks
> Hari
>
> --
> Hari Shreedharan
>
> On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
>
> It seems quite easier than dealing with commons configuration.
>
> I might take a shot at it during the week-end, if I find the time.
>
> About the customisation of the Application class, that's something which
> is already mostly done by
> https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration
> from Zookeeper watcher
>
> It might be a good idea to split this improvement into :
> 1) Selection of an alternative configuration provider
> 2) A zookeeper based configuration provider
>
> If point 1 is done, anyone could use her/his custom configuration provider
> by dropping the implementation as a third party jar and then selecting it
> with the help of the command line.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <de...@gmail.com>wrote:
>
> Thanks for the hints!
>
> The part that you quoted is in fact from the flume wiki and I forgot to
> quote it myself. But the page is quite old so I wasn't sure about the state
> of the feature. I thought maybe commons-configuration was used but wasn't
> be able to confirm it clearly (or more exactly I lost myself in this
> dependency, which might be used by one of my current projects somehow).
>
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
>
> Configuration Provider
>
> Flume NG has a pluggable configuration system called the configuration
> provider. By default, Flume NG ships with a Java property file based
> configuration system that is both simple and easy to generate
> programmatically. Flume OG has a centralized configuration system with
> a master and ZooKeeper for coordination and we recognize this is very
> appealing to some users where as others see it as overhead they simply
> don't want. We opted to make this a pluggable extension point and ship
> a basic implementation that would let many users get started quickly
> and easily. There's almost certainly enough desire for a similar
> implementation to that of Flume OG, but it isn't yet implemented.
> Users may also implement arbitrary plugins to integrate with any type
> of configuration system (JSON files, a shared RDBMS, a central system
> with ZooKeeper, and so forth). We see this as something more
> interesting to system integrators.
>
>
> I will look at the classes you pointed for my own education and I will see
> what I can do.
>
> Thanks a lot.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <br...@cloudera.com> wrote:
>
> Hi,
>
> "Flume NG has a pluggable configuration system called the configuration
> provider."
>
> There is an object hierarchy, but I don't consider it pluggable as there
> is no place to "plugin" your own implementation. I consider these things
> "private" to Flume. Meaning that there is no guaranteed API stability
> between releases. With that caveat, you'd want to implement the following
> abstract class:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>
> e.g
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>
> and then'd you have to customize the Application class to "plugin" your
> own implementation:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>
> Brock
>
>
> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <de...@gmail.com>wrote:
>
> Hi,
>
> Flume NG has a pluggable configuration system called the configuration
> provider. I cloned the source repository and I indeed found it but I am a
> bit lost. Is there some kind of short guide explaining how to write a
> custom configuration provider?
>
> I would like to start with something simple : using a YAML configuration
> file instead of a properties file.
> It should be almost trivial to change the following configuration
>
> *agent1.channels.ch1.type = memory
>
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
>
> agent1.sinks.log-sink1.channel = ch1
> agent1.sinks.log-sink1.type = logger
>
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = log-sink1*
>
> into
>
> agent1:
>
> channels:
> ch1:
> type: memory
>
> sources:
> avroSource1:
> channels: ch1
> type: avro
> bind: 0.0.0.0
> port: 41414
>
> sinks:
> logSink1:
> channel: ch1
> type: logger
>
>
> At least, I know how to convert the latter into the former but I am not
> sure how this transformation can be included into flume-ng using a
> configuration provider.
> I have not been able to find much information about the customisation of
> the configuration provider. Did I miss any documentation?
>
>
> Bertrand
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
>
>
>
>
> --
> Bertrand Dechoux
>
>
>
--
Bertrand Dechoux
Re: Short guide to custom configuration provider?
Posted by Hari Shreedharan <hs...@cloudera.com>.
I actually do like the idea of pluggable config provider. Like you mentioned, it should be possible to hack the Application class, and use a CLI arg to select the config provider. If no CLI arg is provided, just fall back to the current mechanism. If you submit a patch, I'd be glad to review it :-)
Thanks
Hari
--
Hari Shreedharan
On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
> It seems quite easier than dealing with commons configuration.
>
> I might take a shot at it during the week-end, if I find the time.
>
> About the customisation of the Application class, that's something which is already mostly done by
> https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration from Zookeeper watcher
>
> It might be a good idea to split this improvement into :
> 1) Selection of an alternative configuration provider
> 2) A zookeeper based configuration provider
>
> If point 1 is done, anyone could use her/his custom configuration provider by dropping the implementation as a third party jar and then selecting it with the help of the command line.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <dechouxb@gmail.com (mailto:dechouxb@gmail.com)> wrote:
> > Thanks for the hints!
> >
> > The part that you quoted is in fact from the flume wiki and I forgot to quote it myself. But the page is quite old so I wasn't sure about the state of the feature. I thought maybe commons-configuration was used but wasn't be able to confirm it clearly (or more exactly I lost myself in this dependency, which might be used by one of my current projects somehow).
> >
> > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
> >
> > > Configuration Provider
> > >
> > > Flume NG has a pluggable configuration system called the configuration
> > > provider. By default, Flume NG ships with a Java property file based
> > > configuration system that is both simple and easy to generate
> > > programmatically. Flume OG has a centralized configuration system with
> > > a master and ZooKeeper for coordination and we recognize this is very
> > > appealing to some users where as others see it as overhead they simply
> > > don't want. We opted to make this a pluggable extension point and ship
> > > a basic implementation that would let many users get started quickly
> > > and easily. There's almost certainly enough desire for a similar
> > > implementation to that of Flume OG, but it isn't yet implemented.
> > > Users may also implement arbitrary plugins to integrate with any type
> > > of configuration system (JSON files, a shared RDBMS, a central system
> > > with ZooKeeper, and so forth). We see this as something more
> > > interesting to system integrators.
> >
> > I will look at the classes you pointed for my own education and I will see what I can do.
> >
> > Thanks a lot.
> >
> > Bertrand
> >
> >
> > On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <brock@cloudera.com (mailto:brock@cloudera.com)> wrote:
> > > Hi,
> > >
> > > "Flume NG has a pluggable configuration system called the configuration provider."
> > >
> > > There is an object hierarchy, but I don't consider it pluggable as there is no place to "plugin" your own implementation. I consider these things "private" to Flume. Meaning that there is no guaranteed API stability between releases. With that caveat, you'd want to implement the following abstract class:
> > >
> > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
> > >
> > > e.g https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
> > >
> > > and then'd you have to customize the Application class to "plugin" your own implementation:
> > >
> > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
> > >
> > > Brock
> > >
> > >
> > > On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <dechouxb@gmail.com (mailto:dechouxb@gmail.com)> wrote:
> > > > Hi,
> > > >
> > > > Flume NG has a pluggable configuration system called the configuration provider. I cloned the source repository and I indeed found it but I am a bit lost. Is there some kind of short guide explaining how to write a custom configuration provider?
> > > >
> > > > I would like to start with something simple : using a YAML configuration file instead of a properties file.
> > > > It should be almost trivial to change the following configuration
> > > >
> > > > agent1.channels.ch1.type = memory
> > > >
> > > > agent1.sources.avro-source1.channels = ch1
> > > > agent1.sources.avro-source1.type = avro
> > > > agent1.sources.avro-source1.bind = 0.0.0.0
> > > > agent1.sources.avro-source1.port = 41414
> > > >
> > > > agent1.sinks.log-sink1.channel = ch1
> > > > agent1.sinks.log-sink1.type = logger
> > > >
> > > > agent1.channels = ch1
> > > > agent1.sources = avro-source1
> > > > agent1.sinks = log-sink1
> > > >
> > > > into
> > > >
> > > > agent1:
> > > >
> > > > channels:
> > > > ch1:
> > > > type: memory
> > > >
> > > > sources:
> > > > avroSource1:
> > > > channels: ch1
> > > > type: avro
> > > > bind: 0.0.0.0
> > > > port: 41414
> > > >
> > > > sinks:
> > > > logSink1:
> > > > channel: ch1
> > > > type: logger
> > > >
> > > >
> > > > At least, I know how to convert the latter into the former but I am not sure how this transformation can be included into flume-ng using a configuration provider.
> > > > I have not been able to find much information about the customisation of the configuration provider. Did I miss any documentation?
> > > >
> > > >
> > > > Bertrand
> > >
> > >
> > >
> > > --
> > > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
>
>
> --
> Bertrand Dechoux
Re: Short guide to custom configuration provider?
Posted by Bertrand Dechoux <de...@gmail.com>.
It seems quite easier than dealing with commons configuration.
I might take a shot at it during the week-end, if I find the time.
About the customisation of the Application class, that's something which is
already mostly done by
https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration
from Zookeeper watcher
It might be a good idea to split this improvement into :
1) Selection of an alternative configuration provider
2) A zookeeper based configuration provider
If point 1 is done, anyone could use her/his custom configuration provider
by dropping the implementation as a third party jar and then selecting it
with the help of the command line.
Bertrand
On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <de...@gmail.com>wrote:
> Thanks for the hints!
>
> The part that you quoted is in fact from the flume wiki and I forgot to
> quote it myself. But the page is quite old so I wasn't sure about the state
> of the feature. I thought maybe commons-configuration was used but wasn't
> be able to confirm it clearly (or more exactly I lost myself in this
> dependency, which might be used by one of my current projects somehow).
>
> https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
>
> Configuration Provider
>>
>> Flume NG has a pluggable configuration system called the configuration
>> provider. By default, Flume NG ships with a Java property file based
>> configuration system that is both simple and easy to generate
>> programmatically. Flume OG has a centralized configuration system with
>> a master and ZooKeeper for coordination and we recognize this is very
>> appealing to some users where as others see it as overhead they simply
>> don't want. We opted to make this a pluggable extension point and ship
>> a basic implementation that would let many users get started quickly
>> and easily. There's almost certainly enough desire for a similar
>> implementation to that of Flume OG, but it isn't yet implemented.
>> Users may also implement arbitrary plugins to integrate with any type
>> of configuration system (JSON files, a shared RDBMS, a central system
>> with ZooKeeper, and so forth). We see this as something more
>> interesting to system integrators.
>>
>
> I will look at the classes you pointed for my own education and I will see
> what I can do.
>
> Thanks a lot.
>
> Bertrand
>
>
> On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <br...@cloudera.com> wrote:
>
>> Hi,
>>
>> "Flume NG has a pluggable configuration system called the configuration
>> provider."
>>
>> There is an object hierarchy, but I don't consider it pluggable as there
>> is no place to "plugin" your own implementation. I consider these things
>> "private" to Flume. Meaning that there is no guaranteed API stability
>> between releases. With that caveat, you'd want to implement the following
>> abstract class:
>>
>>
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>>
>> e.g
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>>
>> and then'd you have to customize the Application class to "plugin" your
>> own implementation:
>>
>>
>> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>>
>> Brock
>>
>>
>> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <de...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> Flume NG has a pluggable configuration system called the configuration
>>> provider. I cloned the source repository and I indeed found it but I am a
>>> bit lost. Is there some kind of short guide explaining how to write a
>>> custom configuration provider?
>>>
>>> I would like to start with something simple : using a YAML configuration
>>> file instead of a properties file.
>>> It should be almost trivial to change the following configuration
>>>
>>> *agent1.channels.ch1.type = memory
>>>
>>> agent1.sources.avro-source1.channels = ch1
>>> agent1.sources.avro-source1.type = avro
>>> agent1.sources.avro-source1.bind = 0.0.0.0
>>> agent1.sources.avro-source1.port = 41414
>>>
>>> agent1.sinks.log-sink1.channel = ch1
>>> agent1.sinks.log-sink1.type = logger
>>>
>>> agent1.channels = ch1
>>> agent1.sources = avro-source1
>>> agent1.sinks = log-sink1*
>>>
>>> into
>>>
>>> agent1:
>>>
>>> channels:
>>> ch1:
>>> type: memory
>>>
>>> sources:
>>> avroSource1:
>>> channels: ch1
>>> type: avro
>>> bind: 0.0.0.0
>>> port: 41414
>>>
>>> sinks:
>>> logSink1:
>>> channel: ch1
>>> type: logger
>>>
>>>
>>> At least, I know how to convert the latter into the former but I am not
>>> sure how this transformation can be included into flume-ng using a
>>> configuration provider.
>>> I have not been able to find much information about the customisation of
>>> the configuration provider. Did I miss any documentation?
>>>
>>>
>>> Bertrand
>>>
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>>
>
>
--
Bertrand Dechoux
Re: Short guide to custom configuration provider?
Posted by Bertrand Dechoux <de...@gmail.com>.
Thanks for the hints!
The part that you quoted is in fact from the flume wiki and I forgot to
quote it myself. But the page is quite old so I wasn't sure about the state
of the feature. I thought maybe commons-configuration was used but wasn't
be able to confirm it clearly (or more exactly I lost myself in this
dependency, which might be used by one of my current projects somehow).
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG
Configuration Provider
> Flume NG has a pluggable configuration system called the configuration
> provider. By default, Flume NG ships with a Java property file based
> configuration system that is both simple and easy to generate
> programmatically. Flume OG has a centralized configuration system with
> a master and ZooKeeper for coordination and we recognize this is very
> appealing to some users where as others see it as overhead they simply
> don't want. We opted to make this a pluggable extension point and ship
> a basic implementation that would let many users get started quickly
> and easily. There's almost certainly enough desire for a similar
> implementation to that of Flume OG, but it isn't yet implemented.
> Users may also implement arbitrary plugins to integrate with any type
> of configuration system (JSON files, a shared RDBMS, a central system
> with ZooKeeper, and so forth). We see this as something more
> interesting to system integrators.
>
I will look at the classes you pointed for my own education and I will see
what I can do.
Thanks a lot.
Bertrand
On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <br...@cloudera.com> wrote:
> Hi,
>
> "Flume NG has a pluggable configuration system called the configuration
> provider."
>
> There is an object hierarchy, but I don't consider it pluggable as there
> is no place to "plugin" your own implementation. I consider these things
> "private" to Flume. Meaning that there is no guaranteed API stability
> between releases. With that caveat, you'd want to implement the following
> abstract class:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
>
> e.g
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
>
> and then'd you have to customize the Application class to "plugin" your
> own implementation:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
>
> Brock
>
>
> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <de...@gmail.com>wrote:
>
>> Hi,
>>
>> Flume NG has a pluggable configuration system called the configuration
>> provider. I cloned the source repository and I indeed found it but I am a
>> bit lost. Is there some kind of short guide explaining how to write a
>> custom configuration provider?
>>
>> I would like to start with something simple : using a YAML configuration
>> file instead of a properties file.
>> It should be almost trivial to change the following configuration
>>
>> *agent1.channels.ch1.type = memory
>>
>> agent1.sources.avro-source1.channels = ch1
>> agent1.sources.avro-source1.type = avro
>> agent1.sources.avro-source1.bind = 0.0.0.0
>> agent1.sources.avro-source1.port = 41414
>>
>> agent1.sinks.log-sink1.channel = ch1
>> agent1.sinks.log-sink1.type = logger
>>
>> agent1.channels = ch1
>> agent1.sources = avro-source1
>> agent1.sinks = log-sink1*
>>
>> into
>>
>> agent1:
>>
>> channels:
>> ch1:
>> type: memory
>>
>> sources:
>> avroSource1:
>> channels: ch1
>> type: avro
>> bind: 0.0.0.0
>> port: 41414
>>
>> sinks:
>> logSink1:
>> channel: ch1
>> type: logger
>>
>>
>> At least, I know how to convert the latter into the former but I am not
>> sure how this transformation can be included into flume-ng using a
>> configuration provider.
>> I have not been able to find much information about the customisation of
>> the configuration provider. Did I miss any documentation?
>>
>>
>> Bertrand
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>
Re: Short guide to custom configuration provider?
Posted by Brock Noland <br...@cloudera.com>.
Hi,
"Flume NG has a pluggable configuration system called the configuration
provider."
There is an object hierarchy, but I don't consider it pluggable as there is
no place to "plugin" your own implementation. I consider these things
"private" to Flume. Meaning that there is no guaranteed API stability
between releases. With that caveat, you'd want to implement the following
abstract class:
https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java
e.g
https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java
and then'd you have to customize the Application class to "plugin" your own
implementation:
https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
Brock
On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <de...@gmail.com>wrote:
> Hi,
>
> Flume NG has a pluggable configuration system called the configuration
> provider. I cloned the source repository and I indeed found it but I am a
> bit lost. Is there some kind of short guide explaining how to write a
> custom configuration provider?
>
> I would like to start with something simple : using a YAML configuration
> file instead of a properties file.
> It should be almost trivial to change the following configuration
>
> *agent1.channels.ch1.type = memory
>
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
>
> agent1.sinks.log-sink1.channel = ch1
> agent1.sinks.log-sink1.type = logger
>
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = log-sink1*
>
> into
>
> agent1:
>
> channels:
> ch1:
> type: memory
>
> sources:
> avroSource1:
> channels: ch1
> type: avro
> bind: 0.0.0.0
> port: 41414
>
> sinks:
> logSink1:
> channel: ch1
> type: logger
>
>
> At least, I know how to convert the latter into the former but I am not
> sure how this transformation can be included into flume-ng using a
> configuration provider.
> I have not been able to find much information about the customisation of
> the configuration provider. Did I miss any documentation?
>
>
> Bertrand
>
--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org