You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Connor Woodson <cw...@gmail.com> on 2013/03/15 01:09:52 UTC
Parameters in Configuration File
Does the Flume configuration file support parameters, for instance in the
way that you can specify them in a log4j conf file?
- Connor
Re: Parameters in Configuration File
Posted by Connor Woodson <cw...@gmail.com>.
There is some work on modifying the configuration system to make Flume work
with Zookeeper. I believe the way that's being done is by allowing for some
form of pluggable configuration system - you choose between the default
properties file or from something else. I'll look for the JIRA later; but
if you want, it should be possible to make a configuration mechanism to use
Spring instead of the properties file (I don't know enough about Spring to
say more about how it could be done).
Essentially, it should be possible to do, if you or someone else wants to
invest the time in making it work.
- Connor
On Fri, Mar 15, 2013 at 3:07 PM, Kenison, Matt <Ma...@disney.com>wrote:
> I understand you have to continue to support existing clients. However, I
> don't know that total backwards compatibility would be desireable in the
> long run. If you want to support both the existing property configuration
> and a new one, one way would be to expose setters for each attribute of
> selectors, sinks, etc. This would let them be set directly, and allow the
> configuration to be moved to a separate factory class which would be used
> by the existing property files.
>
> I have not dealt in detail with the configuration, though, so I'm not sure
> if there's something I'm overlooking.
>
>
> From: Mike Percy <mp...@apache.org>
> Reply-To: "user@flume.apache.org" <us...@flume.apache.org>
> Date: Fri, 15 Mar 2013 11:37:42 -0700
>
> To: "user@flume.apache.org" <us...@flume.apache.org>
> Subject: Re: Parameters in Configuration File
>
> Hi Matt,
> Properties files were the simple way to get started, and now that Flume is
> on version 1.3.1 we can't just abandon the properties file format.
>
> I'm open to suggestions for adopting more full featured configuration
> functionality while maintaining backwards compatibility with the existing
> config files.
>
> Regards,
> Mike
>
>
> On Fri, Mar 15, 2013 at 10:55 AM, Kenison, Matt <Ma...@disney.com>wrote:
>
>> I'm sure this discussion has happened already, but is there a reason
>> Spring (or something similar) isn't used for configuration, rather than the
>> property files? It seems like there is a lot of boilerplate logic in the
>> various implementations of Configurable.configure(Context) that is just
>> completely unnecessary. In addition, you wouldn't have to spend time
>> building something like property substitution that's been done many times
>> before.
>>
>>
>> From: Mike Percy <mp...@apache.org>
>> Reply-To: "user@flume.apache.org" <us...@flume.apache.org>
>> Date: Fri, 15 Mar 2013 10:27:00 -0700
>> To: "user@flume.apache.org" <us...@flume.apache.org>
>> Subject: Re: Parameters in Configuration File
>>
>> Hey Connor,
>> Take a look at the discussion @
>> https://issues.apache.org/jira/browse/FLUME-1941
>>
>> I you want to help work on this you are more than welcome to :)
>>
>> Regards,
>> Mike
>>
>>
>>
>> On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <cw...@gmail.com>wrote:
>>
>>> Does the Flume configuration file support parameters, for instance in
>>> the way that you can specify them in a log4j conf file?
>>>
>>> - Connor
>>>
>>
>>
>
Re: Parameters in Configuration File
Posted by "Kenison, Matt" <Ma...@disney.com>.
I understand you have to continue to support existing clients. However, I don't know that total backwards compatibility would be desireable in the long run. If you want to support both the existing property configuration and a new one, one way would be to expose setters for each attribute of selectors, sinks, etc. This would let them be set directly, and allow the configuration to be moved to a separate factory class which would be used by the existing property files.
I have not dealt in detail with the configuration, though, so I'm not sure if there's something I'm overlooking.
From: Mike Percy <mp...@apache.org>>
Reply-To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Date: Fri, 15 Mar 2013 11:37:42 -0700
To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Subject: Re: Parameters in Configuration File
Hi Matt,
Properties files were the simple way to get started, and now that Flume is on version 1.3.1 we can't just abandon the properties file format.
I'm open to suggestions for adopting more full featured configuration functionality while maintaining backwards compatibility with the existing config files.
Regards,
Mike
On Fri, Mar 15, 2013 at 10:55 AM, Kenison, Matt <Ma...@disney.com>> wrote:
I'm sure this discussion has happened already, but is there a reason Spring (or something similar) isn't used for configuration, rather than the property files? It seems like there is a lot of boilerplate logic in the various implementations of Configurable.configure(Context) that is just completely unnecessary. In addition, you wouldn't have to spend time building something like property substitution that's been done many times before.
From: Mike Percy <mp...@apache.org>>
Reply-To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Date: Fri, 15 Mar 2013 10:27:00 -0700
To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Subject: Re: Parameters in Configuration File
Hey Connor,
Take a look at the discussion @ https://issues.apache.org/jira/browse/FLUME-1941
I you want to help work on this you are more than welcome to :)
Regards,
Mike
On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <cw...@gmail.com>> wrote:
Does the Flume configuration file support parameters, for instance in the way that you can specify them in a log4j conf file?
- Connor
Re: Parameters in Configuration File
Posted by Mike Percy <mp...@apache.org>.
Hi Matt,
Properties files were the simple way to get started, and now that Flume is
on version 1.3.1 we can't just abandon the properties file format.
I'm open to suggestions for adopting more full featured configuration
functionality while maintaining backwards compatibility with the existing
config files.
Regards,
Mike
On Fri, Mar 15, 2013 at 10:55 AM, Kenison, Matt <Ma...@disney.com>wrote:
> I'm sure this discussion has happened already, but is there a reason
> Spring (or something similar) isn't used for configuration, rather than the
> property files? It seems like there is a lot of boilerplate logic in the
> various implementations of Configurable.configure(Context) that is just
> completely unnecessary. In addition, you wouldn't have to spend time
> building something like property substitution that's been done many times
> before.
>
>
> From: Mike Percy <mp...@apache.org>
> Reply-To: "user@flume.apache.org" <us...@flume.apache.org>
> Date: Fri, 15 Mar 2013 10:27:00 -0700
> To: "user@flume.apache.org" <us...@flume.apache.org>
> Subject: Re: Parameters in Configuration File
>
> Hey Connor,
> Take a look at the discussion @
> https://issues.apache.org/jira/browse/FLUME-1941
>
> I you want to help work on this you are more than welcome to :)
>
> Regards,
> Mike
>
>
>
> On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <cw...@gmail.com>wrote:
>
>> Does the Flume configuration file support parameters, for instance in the
>> way that you can specify them in a log4j conf file?
>>
>> - Connor
>>
>
>
Re: Parameters in Configuration File
Posted by "Kenison, Matt" <Ma...@disney.com>.
I'm sure this discussion has happened already, but is there a reason Spring (or something similar) isn't used for configuration, rather than the property files? It seems like there is a lot of boilerplate logic in the various implementations of Configurable.configure(Context) that is just completely unnecessary. In addition, you wouldn't have to spend time building something like property substitution that's been done many times before.
From: Mike Percy <mp...@apache.org>>
Reply-To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Date: Fri, 15 Mar 2013 10:27:00 -0700
To: "user@flume.apache.org<ma...@flume.apache.org>" <us...@flume.apache.org>>
Subject: Re: Parameters in Configuration File
Hey Connor,
Take a look at the discussion @ https://issues.apache.org/jira/browse/FLUME-1941
I you want to help work on this you are more than welcome to :)
Regards,
Mike
On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <cw...@gmail.com>> wrote:
Does the Flume configuration file support parameters, for instance in the way that you can specify them in a log4j conf file?
- Connor
Re: Parameters in Configuration File
Posted by Mike Percy <mp...@apache.org>.
Hey Connor,
Take a look at the discussion @
https://issues.apache.org/jira/browse/FLUME-1941
I you want to help work on this you are more than welcome to :)
Regards,
Mike
On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <cw...@gmail.com>wrote:
> Does the Flume configuration file support parameters, for instance in the
> way that you can specify them in a log4j conf file?
>
> - Connor
>