Sending custom config values from nimbus to components

[Matt Lowe <ma...@gmail.com>]

Hello!

When you deploy a topology from nimbus, you can see the topology config
that will be sent to the other components.
These are custom values from default.yaml that have been altered.

Is there a way to add NON topology related config values?
I have a config file that my nimbus reads, and I want to distribute the
values FROM the nimbus.

Any ideas?

//Matt

Re: Sending custom config values from nimbus to components

[Punit Tiwan <ti...@coviam.com>]

In SubmitTopology method, you can pass custom configurations and those will be passed on to bolts as Map in "prepare" method.
So while submitting a topology, you can pass your properties file path as argument and in your main class, you can read those properties and pass on to bolts.

Regards,
Punit Tiwan

> On 26-Jan-2017, at 11:09 AM, Joaquin Menchaca <jm...@gobalto.com> wrote:
> 
> For clarification, you are not submitting "from" a nimbus, you are submitting to "nimbus" scheduler.  You can do this from any system that has storm installed, provided storm.yaml configured with nimbus, and you have access to cluster systems.  
> 
> If you want to save values that the topology can use, you can have a properties file with your data that is bundled into your jar.  If these are dynamically configured, then this is a deploy time config, as the jars are immutable once deployed.  
> 
> If you need to maintain state configured by an external source, then you can use service discovery like etcd, consul, or zookeeper.  Recommend avoiding that path, as it is complicated, and use deploy time state config. 
> 
> For my environment, I use a staging deploy container, and use ansible to template up a config, orchestrate building a jar, and submitting the jar remotely from the container to a nimbus in the cluster.  The container encapsulates all the config, and library, vm versions required.
> 
> - Joaquin Menchaca
> 
> On Jan 9, 2017 10:10 AM, "Matt Lowe" <ma...@gmail.com> wrote:
> 
> 
> Hello!
> 
> When you deploy a topology from nimbus, you can see the topology config that will be sent to the other components. 
> These are custom values from default.yaml that have been altered.
> 
> Is there a way to add NON topology related config values?
> I have a config file that my nimbus reads, and I want to distribute the values FROM the nimbus.
> 
> Any ideas?
> 
> //Matt
> 

Re: Sending custom config values from nimbus to components

[Joaquin Menchaca <jm...@gobalto.com>]

For clarification, you are not submitting "from" a nimbus, you are
submitting to "nimbus" scheduler.  You can do this from any system that has
storm installed, provided storm.yaml configured with nimbus, and you have
access to cluster systems.

If you want to save values that the topology can use, you can have a
properties file with your data that is bundled into your jar.  If these are
dynamically configured, then this is a deploy time config, as the jars are
immutable once deployed.

If you need to maintain state configured by an external source, then you
can use service discovery like etcd, consul, or zookeeper.  Recommend
avoiding that path, as it is complicated, and use deploy time state config.

For my environment, I use a staging deploy container, and use ansible to
template up a config, orchestrate building a jar, and submitting the jar
remotely from the container to a nimbus in the cluster.  The container
encapsulates all the config, and library, vm versions required.

- Joaquin Menchaca

On Jan 9, 2017 10:10 AM, "Matt Lowe" <ma...@gmail.com> wrote:



Hello!

When you deploy a topology from nimbus, you can see the topology config
that will be sent to the other components.
These are custom values from default.yaml that have been altered.

Is there a way to add NON topology related config values?
I have a config file that my nimbus reads, and I want to distribute the
values FROM the nimbus.

Any ideas?

//Matt