You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by "King, Craig A." <CR...@leidos.com> on 2014/05/13 17:14:03 UTC

Re: Dynamic Properties Revisited

I submitted this question back in March, but did not get any responses. Since a little time has passed, and there are a few more folks on the mail list, I thought I would pop it back up again.

Thanks in advance,

Craig

On Mar 14, 2014, at 10:26 AM, King, Craig A. <CR...@leidos.com>> wrote:

This topic was covered before, but it does not entirely fit my use case.

I am looking for some best practices, or ideas on how to manage user names/passwords and other properties that can change at any time.

The previous discussion revolved around "external" configuration at submission time, and can be found here:
http://grokbase.com/t/gg/storm-user/134r0rbepz/submitting-a-jar-with-external-config

For background, I am doing an analysis of Storm for a DoD/Navy project. Within the Navy there are IA (Information Assurance) rules that govern password changes (such as passwords must change every 30 or 45 days etc.) We also need to design the administration of the system for 19 year old sailors with a few months training.

In order to manage the properties, there would be some web based UI that would allow the admin to update passwords and hit a save button. No file editing or logging into Nimbus to change configuration files.

The updated passwords (and other changed properties) should become immediately available to all currently running topologies. There could be dozens or even hundreds of topologies running, so killing and resubmitting with new properties is not really an option.

I have a couple of ideas, but I am a storm newbie so I don't know the feasibility...
1) have the spouts monitor a property server for changes and push configuration (would require that all bolts get these streams.)
2) have each spout an bolt monitor the said property server.
3) use Messaging and have spouts/bolts subscribe to a configuration topic.

All ideas are welcome. Thanks in advance.

Craig

RE: Dynamic Properties Revisited

Posted by "King, Craig A." <CR...@leidos.com>.

David,

Sorry... missed the Zookeeper part of your response.  I will look into that.   I went to the archaius wiki, and the first configuration source that it mentioned was JDBC... so I got hung up on that.  I will check out the Zookeeper module.

Craig

________________________________
From: user-return-2171-CRAIG.A.KING=leidos.com@storm.incubator.apache.org [user-return-2171-CRAIG.A.KING=leidos.com@storm.incubator.apache.org] on behalf of David Miller [david.miller@m-square.com.au]
Sent: Wednesday, May 14, 2014 1:50 AM
To: user@storm.incubator.apache.org
Cc: user@storm.incubator.apache.org
Subject: Re: Dynamic Properties Revisited

Not sure what you mean by jdbc when I suggested using the zookeeper archaius module.

Setting it up is simple, there's a few different ways all covered in the docs

On 14/05/2014, at 12:31 PM, "King, Craig A." <CR...@leidos.com>> wrote:

David,

So the question that I have with archaius is how you go about bootstrapping the credentials to connect to a jdbc based configuration?  I could make a service out of it, but then the bolt would have to authenticate to the service…

Even if we used certificates, we would need to disseminate them throughout the topology.

Aaron,

How would a bolt connect back to Zookeeper to get configuration information?

Thanks in advance,

Craig

On May 13, 2014, at 4:24 PM, David Miller <da...@m-square.com.au>> wrote:

We use archaius with the zookeeper module for this

https://github.com/Netflix/archaius/wiki

On Wed, May 14, 2014 at 1:18 AM, Aaron Zimmerman <az...@sproutsocial.com>> wrote:
I would put it in zookeeper, especially since that's already a dependency.

On Tue, May 13, 2014 at 10:14 AM, King, Craig A. <CR...@leidos.com>> wrote:
I submitted this question back in March, but did not get any responses.  Since a little time has passed, and there are a few more folks on the mail list, I thought I would pop it back up again.

Thanks in advance,

Craig

On Mar 14, 2014, at 10:26 AM, King, Craig A. <CR...@leidos.com>> wrote:

This topic was covered before, but it does not entirely fit my use case.

I am looking for some best practices, or ideas on how to manage user names/passwords and other properties that can change at any time.

The previous discussion revolved around "external" configuration at submission time, and can be found here:
http://grokbase.com/t/gg/storm-user/134r0rbepz/submitting-a-jar-with-external-config

For background, I am doing an analysis of Storm for a DoD/Navy project.  Within the Navy there are IA (Information Assurance) rules that govern password changes (such as passwords must change every 30 or 45 days etc.) We also need to design the administration of the system for 19 year old sailors with a few months training.

In order to manage the properties, there would be some web based UI that would allow the admin to update passwords and hit a save button. No file editing or logging into Nimbus to change configuration files.

The updated passwords (and other changed properties) should become immediately available to all currently running topologies.  There could be dozens or even hundreds of topologies running, so killing and resubmitting with new properties is not really an option.

I have a couple of ideas, but I am a storm newbie so I don't know the feasibility...
1) have the spouts monitor a property server for changes and push configuration (would require that all bolts get these streams.)
2) have each spout an bolt monitor the said property server.
3) use Messaging and have spouts/bolts subscribe to a configuration topic.

All ideas are welcome.  Thanks in advance.

Craig

Re: Dynamic Properties Revisited

Posted by David Miller <da...@m-square.com.au>.

Not sure what you mean by jdbc when I suggested using the zookeeper archaius module.

Setting it up is simple, there's a few different ways all covered in the docs

On 14/05/2014, at 12:31 PM, "King, Craig A." <CR...@leidos.com> wrote:

> David,
> 
> So the question that I have with archaius is how you go about bootstrapping the credentials to connect to a jdbc based configuration?  I could make a service out of it, but then the bolt would have to authenticate to the service…  
> 
> Even if we used certificates, we would need to disseminate them throughout the topology.
> 
> Aaron,
> 
> How would a bolt connect back to Zookeeper to get configuration information?
> 
> Thanks in advance,
> 
> Craig
> 
> On May 13, 2014, at 4:24 PM, David Miller <da...@m-square.com.au> wrote:
> 
>> We use archaius with the zookeeper module for this
>> 
>> https://github.com/Netflix/archaius/wiki
>> 
>> 
>> On Wed, May 14, 2014 at 1:18 AM, Aaron Zimmerman <az...@sproutsocial.com> wrote:
>>> I would put it in zookeeper, especially since that's already a dependency.  
>>> 
>>> 
>>> On Tue, May 13, 2014 at 10:14 AM, King, Craig A. <CR...@leidos.com> wrote:
>>>> I submitted this question back in March, but did not get any responses.  Since a little time has passed, and there are a few more folks on the mail list, I thought I would pop it back up again.
>>>> 
>>>> Thanks in advance,
>>>> 
>>>> Craig
>>>> 
>>>> On Mar 14, 2014, at 10:26 AM, King, Craig A. <CR...@leidos.com> wrote:
>>>> 
>>>>> This topic was covered before, but it does not entirely fit my use case.
>>>>> 
>>>>> I am looking for some best practices, or ideas on how to manage user names/passwords and other properties that can change at any time.
>>>>> 
>>>>> The previous discussion revolved around "external" configuration at submission time, and can be found here:
>>>>> http://grokbase.com/t/gg/storm-user/134r0rbepz/submitting-a-jar-with-external-config
>>>>> 
>>>>> For background, I am doing an analysis of Storm for a DoD/Navy project.  Within the Navy there are IA (Information Assurance) rules that govern password changes (such as passwords must change every 30 or 45 days etc.) We also need to design the administration of the system for 19 year old sailors with a few months training.
>>>>> 
>>>>> In order to manage the properties, there would be some web based UI that would allow the admin to update passwords and hit a save button. No file editing or logging into Nimbus to change configuration files.
>>>>> 
>>>>> The updated passwords (and other changed properties) should become immediately available to all currently running topologies.  There could be dozens or even hundreds of topologies running, so killing and resubmitting with new properties is not really an option.
>>>>> 
>>>>> I have a couple of ideas, but I am a storm newbie so I don't know the feasibility...  
>>>>> 1) have the spouts monitor a property server for changes and push configuration (would require that all bolts get these streams.)  
>>>>> 2) have each spout an bolt monitor the said property server. 
>>>>> 3) use Messaging and have spouts/bolts subscribe to a configuration topic.
>>>>> 
>>>>> All ideas are welcome.  Thanks in advance.
>>>>> 
>>>>> Craig
>

Re: Dynamic Properties Revisited

Posted by "King, Craig A." <CR...@leidos.com>.

David,

So the question that I have with archaius is how you go about bootstrapping the credentials to connect to a jdbc based configuration? I could make a service out of it, but then the bolt would have to authenticate to the service…

Even if we used certificates, we would need to disseminate them throughout the topology.

Aaron,

How would a bolt connect back to Zookeeper to get configuration information?

Thanks in advance,

Craig

On May 13, 2014, at 4:24 PM, David Miller <da...@m-square.com.au>> wrote:

We use archaius with the zookeeper module for this

https://github.com/Netflix/archaius/wiki

On Wed, May 14, 2014 at 1:18 AM, Aaron Zimmerman <az...@sproutsocial.com>> wrote:
I would put it in zookeeper, especially since that's already a dependency.

On Tue, May 13, 2014 at 10:14 AM, King, Craig A. <CR...@leidos.com>> wrote:
I submitted this question back in March, but did not get any responses. Since a little time has passed, and there are a few more folks on the mail list, I thought I would pop it back up again.

Thanks in advance,

Craig

On Mar 14, 2014, at 10:26 AM, King, Craig A. <CR...@leidos.com>> wrote:

This topic was covered before, but it does not entirely fit my use case.

I am looking for some best practices, or ideas on how to manage user names/passwords and other properties that can change at any time.

The previous discussion revolved around "external" configuration at submission time, and can be found here:
http://grokbase.com/t/gg/storm-user/134r0rbepz/submitting-a-jar-with-external-config

All ideas are welcome. Thanks in advance.

Craig

Re: Dynamic Properties Revisited

Posted by David Miller <da...@m-square.com.au>.

We use archaius with the zookeeper module for this

https://github.com/Netflix/archaius/wiki


On Wed, May 14, 2014 at 1:18 AM, Aaron Zimmerman <
azimmerman@sproutsocial.com> wrote:

> I would put it in zookeeper, especially since that's already a dependency.
>
>
>
> On Tue, May 13, 2014 at 10:14 AM, King, Craig A. <CR...@leidos.com>wrote:
>
>>  I submitted this question back in March, but did not get any responses.
>>  Since a little time has passed, and there are a few more folks on the mail
>> list, I thought I would pop it back up again.
>>
>>  Thanks in advance,
>>
>>  Craig
>>
>>  On Mar 14, 2014, at 10:26 AM, King, Craig A. <CR...@leidos.com>
>> wrote:
>>
>>  This topic was covered before, but it does not entirely fit my use case.
>>
>> I am looking for some best practices, or ideas on how to manage user
>> names/passwords and other properties that can change at any time.
>>
>> The previous discussion revolved around "external" configuration at
>> submission time, and can be found here:
>>
>> http://grokbase.com/t/gg/storm-user/134r0rbepz/submitting-a-jar-with-external-config
>>
>> For background, I am doing an analysis of Storm for a DoD/Navy project.
>> Within the Navy there are IA (Information Assurance) rules that govern
>> password changes (such as passwords must change every 30 or 45 days etc.)
>> We also need to design the administration of the system for 19 year old
>> sailors with a few months training.
>>
>> In order to manage the properties, there would be some web based UI that
>> would allow the admin to update passwords and hit a save button. No file
>> editing or logging into Nimbus to change configuration files.
>>
>> The updated passwords (and other changed properties) should become
>> immediately available to all currently running topologies.  There could be
>> dozens or even hundreds of topologies running, so killing and resubmitting
>> with new properties is not really an option.
>>
>> I have a couple of ideas, but I am a storm newbie so I don't know the
>> feasibility...
>> 1) have the spouts monitor a property server for changes and push
>> configuration (would require that all bolts get these streams.)
>> 2) have each spout an bolt monitor the said property server.
>> 3) use Messaging and have spouts/bolts subscribe to a configuration topic.
>>
>> All ideas are welcome.  Thanks in advance.
>>
>> Craig
>>
>>
>>
>

Re: Dynamic Properties Revisited

Posted by Aaron Zimmerman <az...@sproutsocial.com>.

I would put it in zookeeper, especially since that's already a dependency.


On Tue, May 13, 2014 at 10:14 AM, King, Craig A. <CR...@leidos.com>wrote:

>  I submitted this question back in March, but did not get any responses.
>  Since a little time has passed, and there are a few more folks on the mail
> list, I thought I would pop it back up again.
>
>  Thanks in advance,
>
>  Craig
>
>  On Mar 14, 2014, at 10:26 AM, King, Craig A. <CR...@leidos.com>
> wrote:
>
>  This topic was covered before, but it does not entirely fit my use case.
>
> I am looking for some best practices, or ideas on how to manage user
> names/passwords and other properties that can change at any time.
>
> The previous discussion revolved around "external" configuration at
> submission time, and can be found here:
>
> http://grokbase.com/t/gg/storm-user/134r0rbepz/submitting-a-jar-with-external-config
>
> For background, I am doing an analysis of Storm for a DoD/Navy project.
> Within the Navy there are IA (Information Assurance) rules that govern
> password changes (such as passwords must change every 30 or 45 days etc.)
> We also need to design the administration of the system for 19 year old
> sailors with a few months training.
>
> In order to manage the properties, there would be some web based UI that
> would allow the admin to update passwords and hit a save button. No file
> editing or logging into Nimbus to change configuration files.
>
> The updated passwords (and other changed properties) should become
> immediately available to all currently running topologies.  There could be
> dozens or even hundreds of topologies running, so killing and resubmitting
> with new properties is not really an option.
>
> I have a couple of ideas, but I am a storm newbie so I don't know the
> feasibility...
> 1) have the spouts monitor a property server for changes and push
> configuration (would require that all bolts get these streams.)
> 2) have each spout an bolt monitor the said property server.
> 3) use Messaging and have spouts/bolts subscribe to a configuration topic.
>
> All ideas are welcome.  Thanks in advance.
>
> Craig
>
>
>