You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Erik Weathers <ew...@groupon.com> on 2014/12/05 03:26:56 UTC
storm.yaml config: change on nimbus or worker machines?
Can someone please clarify for me how you properly specify worker-specific
configuration in storm.yaml?
e.g., I'm making a change to use zmq as the transport (due to a tangential
problem with netty), and it's unclear where to make this change:
- storm.messaging.transport: backtype.storm.messaging.zmq
I assumed that I needed to change storm.yaml on the worker machine, but it
seems that the change only shows up in the nimbus UI when I modify
storm.yaml on the nimbus.
So I'm assuming that the config options you specify in storm.yaml on nimbus
propagated to the storm workers, please correct me if that's wrong.
But what if you make a change to the configs in the nimbus and then restart
it with the topology still running? i.e., will that change be propagated
to the existing topology workers somehow? (In the case of changing the
messaging transport, I assume that would mean restarting the workers to use
the alternative transport layer.)
Thanks!
- Erik
P.S., I am well aware of the other stuff you need to do to get zmq working
in storm-0.9.1+; I just don't want to force all of our topologies to
specify the above configuration knob.
P.P.S., I'm using storm-0.9.3
Re: storm.yaml config: change on nimbus or worker machines?
Posted by Susheel Kumar Gadalay <sk...@gmail.com>.
Workers or supervisor machines can have minimum details in storm.yaml.
Nimbus server can have all the configuration details or they can be defaulted.
If you want to overwrite a particular configuration property that
Nimbus gives as
part of topology execution down to all workers,
then it can be overwritten in the topology main class before submit
topology call.
On 12/6/14, Erik Weathers <ew...@groupon.com> wrote:
> Thanks Susheel. Do you know enough about how storm works internally to be
> able to answer my other questions? Specifically, I'd like to know how to
> change the configuration back in the future; would I need to have the
> topologies resubmitted in order to have the configuration passed along? Or
> could I just kill all the workers and when storm creates new workers it
> would propagate the changed settings from the nimbus?
>
> Also, which configuration options are ok to be in the worker host's
> storm.yaml (since from my experiment it seemed insufficient to set this "
> storm.messaging.transport" value in the worker's storm.yaml -- it *seemed*
> this way because the nimbus UI was showing a different value than what I
> configured in the worker storm.yaml)? Where is the distinction drawn
> between a config option that needs to be in the nimbus storm.yaml versus
> the supervisor/worker storm.yaml (a pointer to the code would be awesome)?
> I'm still confused about how configurations are handled in storm and I'm
> worried that there could be different configurations in place on workers
> versus what the nimbus shows.
>
> - Erik
>
> On Fri, Dec 5, 2014 at 12:32 AM, Susheel Kumar Gadalay
> <sk...@gmail.com>
> wrote:
>
>> If required globally for all topologies then set it in Nimbus.
>>
>> It will be carried to all workers when topology submitted.
>>
>> On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
>> > Thanks for the response Susheel. But this setting is required for all
>> > topologies, I need to set it globally so that I don't have to harass
>> every
>> > team into modifying their topologies with that configuration. Also,
>> later
>> > when netty is presumably fixed I don't want to force people to rewrite
>> > their topologies to switch back from zmq to netty.
>> >
>> > - Erik
>> >
>> > On Thu, Dec 4, 2014 at 10:08 PM, Susheel Kumar Gadalay
>> > <sk...@gmail.com>
>> > wrote:
>> >
>> >> If this is required for one topology better to set it in the Config
>> >> object.
>> >>
>> >> Config config = new Config();
>> >> config.put(Config.storm.messaging.transport:
>> >> "backtype.storm.messaging.zmq");
>> >>
>> >>
>> >> On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
>> >> > Can someone please clarify for me how you properly specify
>> >> worker-specific
>> >> > configuration in storm.yaml?
>> >> >
>> >> > e.g., I'm making a change to use zmq as the transport (due to a
>> >> tangential
>> >> > problem with netty), and it's unclear where to make this change:
>> >> >
>> >> > - storm.messaging.transport: backtype.storm.messaging.zmq
>> >> >
>> >> > I assumed that I needed to change storm.yaml on the worker machine,
>> but
>> >> it
>> >> > seems that the change only shows up in the nimbus UI when I modify
>> >> > storm.yaml on the nimbus.
>> >> >
>> >> > So I'm assuming that the config options you specify in storm.yaml on
>> >> nimbus
>> >> > propagated to the storm workers, please correct me if that's wrong.
>> >> >
>> >> > But what if you make a change to the configs in the nimbus and then
>> >> restart
>> >> > it with the topology still running? i.e., will that change be
>> >> > propagated
>> >> > to the existing topology workers somehow? (In the case of changing
>> the
>> >> > messaging transport, I assume that would mean restarting the workers
>> to
>> >> use
>> >> > the alternative transport layer.)
>> >> >
>> >> > Thanks!
>> >> >
>> >> > - Erik
>> >> >
>> >> > P.S., I am well aware of the other stuff you need to do to get zmq
>> >> working
>> >> > in storm-0.9.1+; I just don't want to force all of our topologies to
>> >> > specify the above configuration knob.
>> >> >
>> >> > P.P.S., I'm using storm-0.9.3
>> >> >
>> >>
>> >
>>
>
Re: storm.yaml config: change on nimbus or worker machines?
Posted by Erik Weathers <ew...@groupon.com>.
Thanks Susheel. Do you know enough about how storm works internally to be
able to answer my other questions? Specifically, I'd like to know how to
change the configuration back in the future; would I need to have the
topologies resubmitted in order to have the configuration passed along? Or
could I just kill all the workers and when storm creates new workers it
would propagate the changed settings from the nimbus?
Also, which configuration options are ok to be in the worker host's
storm.yaml (since from my experiment it seemed insufficient to set this "
storm.messaging.transport" value in the worker's storm.yaml -- it *seemed*
this way because the nimbus UI was showing a different value than what I
configured in the worker storm.yaml)? Where is the distinction drawn
between a config option that needs to be in the nimbus storm.yaml versus
the supervisor/worker storm.yaml (a pointer to the code would be awesome)?
I'm still confused about how configurations are handled in storm and I'm
worried that there could be different configurations in place on workers
versus what the nimbus shows.
- Erik
On Fri, Dec 5, 2014 at 12:32 AM, Susheel Kumar Gadalay <sk...@gmail.com>
wrote:
> If required globally for all topologies then set it in Nimbus.
>
> It will be carried to all workers when topology submitted.
>
> On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
> > Thanks for the response Susheel. But this setting is required for all
> > topologies, I need to set it globally so that I don't have to harass
> every
> > team into modifying their topologies with that configuration. Also,
> later
> > when netty is presumably fixed I don't want to force people to rewrite
> > their topologies to switch back from zmq to netty.
> >
> > - Erik
> >
> > On Thu, Dec 4, 2014 at 10:08 PM, Susheel Kumar Gadalay
> > <sk...@gmail.com>
> > wrote:
> >
> >> If this is required for one topology better to set it in the Config
> >> object.
> >>
> >> Config config = new Config();
> >> config.put(Config.storm.messaging.transport:
> >> "backtype.storm.messaging.zmq");
> >>
> >>
> >> On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
> >> > Can someone please clarify for me how you properly specify
> >> worker-specific
> >> > configuration in storm.yaml?
> >> >
> >> > e.g., I'm making a change to use zmq as the transport (due to a
> >> tangential
> >> > problem with netty), and it's unclear where to make this change:
> >> >
> >> > - storm.messaging.transport: backtype.storm.messaging.zmq
> >> >
> >> > I assumed that I needed to change storm.yaml on the worker machine,
> but
> >> it
> >> > seems that the change only shows up in the nimbus UI when I modify
> >> > storm.yaml on the nimbus.
> >> >
> >> > So I'm assuming that the config options you specify in storm.yaml on
> >> nimbus
> >> > propagated to the storm workers, please correct me if that's wrong.
> >> >
> >> > But what if you make a change to the configs in the nimbus and then
> >> restart
> >> > it with the topology still running? i.e., will that change be
> >> > propagated
> >> > to the existing topology workers somehow? (In the case of changing
> the
> >> > messaging transport, I assume that would mean restarting the workers
> to
> >> use
> >> > the alternative transport layer.)
> >> >
> >> > Thanks!
> >> >
> >> > - Erik
> >> >
> >> > P.S., I am well aware of the other stuff you need to do to get zmq
> >> working
> >> > in storm-0.9.1+; I just don't want to force all of our topologies to
> >> > specify the above configuration knob.
> >> >
> >> > P.P.S., I'm using storm-0.9.3
> >> >
> >>
> >
>
Re: storm.yaml config: change on nimbus or worker machines?
Posted by Susheel Kumar Gadalay <sk...@gmail.com>.
If required globally for all topologies then set it in Nimbus.
It will be carried to all workers when topology submitted.
On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
> Thanks for the response Susheel. But this setting is required for all
> topologies, I need to set it globally so that I don't have to harass every
> team into modifying their topologies with that configuration. Also, later
> when netty is presumably fixed I don't want to force people to rewrite
> their topologies to switch back from zmq to netty.
>
> - Erik
>
> On Thu, Dec 4, 2014 at 10:08 PM, Susheel Kumar Gadalay
> <sk...@gmail.com>
> wrote:
>
>> If this is required for one topology better to set it in the Config
>> object.
>>
>> Config config = new Config();
>> config.put(Config.storm.messaging.transport:
>> "backtype.storm.messaging.zmq");
>>
>>
>> On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
>> > Can someone please clarify for me how you properly specify
>> worker-specific
>> > configuration in storm.yaml?
>> >
>> > e.g., I'm making a change to use zmq as the transport (due to a
>> tangential
>> > problem with netty), and it's unclear where to make this change:
>> >
>> > - storm.messaging.transport: backtype.storm.messaging.zmq
>> >
>> > I assumed that I needed to change storm.yaml on the worker machine, but
>> it
>> > seems that the change only shows up in the nimbus UI when I modify
>> > storm.yaml on the nimbus.
>> >
>> > So I'm assuming that the config options you specify in storm.yaml on
>> nimbus
>> > propagated to the storm workers, please correct me if that's wrong.
>> >
>> > But what if you make a change to the configs in the nimbus and then
>> restart
>> > it with the topology still running? i.e., will that change be
>> > propagated
>> > to the existing topology workers somehow? (In the case of changing the
>> > messaging transport, I assume that would mean restarting the workers to
>> use
>> > the alternative transport layer.)
>> >
>> > Thanks!
>> >
>> > - Erik
>> >
>> > P.S., I am well aware of the other stuff you need to do to get zmq
>> working
>> > in storm-0.9.1+; I just don't want to force all of our topologies to
>> > specify the above configuration knob.
>> >
>> > P.P.S., I'm using storm-0.9.3
>> >
>>
>
Re: storm.yaml config: change on nimbus or worker machines?
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Erik,
I would suggest making that change in storm.yaml across all nimbus and supervisor nodes in your cluster (making it global). IMHO, having different storm.yaml files across the cluster is asking for trouble.
Also, only configuration options that start with “topology.*” are topology-specific. So your teams shouldn’t have to modify their topology configurations.
-Taylor
On Dec 5, 2014, at 3:00 AM, Erik Weathers <ew...@groupon.com> wrote:
> Thanks for the response Susheel. But this setting is required for all topologies, I need to set it globally so that I don't have to harass every team into modifying their topologies with that configuration. Also, later when netty is presumably fixed I don't want to force people to rewrite their topologies to switch back from zmq to netty.
>
> - Erik
>
> On Thu, Dec 4, 2014 at 10:08 PM, Susheel Kumar Gadalay <sk...@gmail.com> wrote:
> If this is required for one topology better to set it in the Config object.
>
> Config config = new Config();
> config.put(Config.storm.messaging.transport: "backtype.storm.messaging.zmq");
>
>
> On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
> > Can someone please clarify for me how you properly specify worker-specific
> > configuration in storm.yaml?
> >
> > e.g., I'm making a change to use zmq as the transport (due to a tangential
> > problem with netty), and it's unclear where to make this change:
> >
> > - storm.messaging.transport: backtype.storm.messaging.zmq
> >
> > I assumed that I needed to change storm.yaml on the worker machine, but it
> > seems that the change only shows up in the nimbus UI when I modify
> > storm.yaml on the nimbus.
> >
> > So I'm assuming that the config options you specify in storm.yaml on nimbus
> > propagated to the storm workers, please correct me if that's wrong.
> >
> > But what if you make a change to the configs in the nimbus and then restart
> > it with the topology still running? i.e., will that change be propagated
> > to the existing topology workers somehow? (In the case of changing the
> > messaging transport, I assume that would mean restarting the workers to use
> > the alternative transport layer.)
> >
> > Thanks!
> >
> > - Erik
> >
> > P.S., I am well aware of the other stuff you need to do to get zmq working
> > in storm-0.9.1+; I just don't want to force all of our topologies to
> > specify the above configuration knob.
> >
> > P.P.S., I'm using storm-0.9.3
> >
>
Re: storm.yaml config: change on nimbus or worker machines?
Posted by Erik Weathers <ew...@groupon.com>.
Thanks for the response Susheel. But this setting is required for all
topologies, I need to set it globally so that I don't have to harass every
team into modifying their topologies with that configuration. Also, later
when netty is presumably fixed I don't want to force people to rewrite
their topologies to switch back from zmq to netty.
- Erik
On Thu, Dec 4, 2014 at 10:08 PM, Susheel Kumar Gadalay <sk...@gmail.com>
wrote:
> If this is required for one topology better to set it in the Config object.
>
> Config config = new Config();
> config.put(Config.storm.messaging.transport:
> "backtype.storm.messaging.zmq");
>
>
> On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
> > Can someone please clarify for me how you properly specify
> worker-specific
> > configuration in storm.yaml?
> >
> > e.g., I'm making a change to use zmq as the transport (due to a
> tangential
> > problem with netty), and it's unclear where to make this change:
> >
> > - storm.messaging.transport: backtype.storm.messaging.zmq
> >
> > I assumed that I needed to change storm.yaml on the worker machine, but
> it
> > seems that the change only shows up in the nimbus UI when I modify
> > storm.yaml on the nimbus.
> >
> > So I'm assuming that the config options you specify in storm.yaml on
> nimbus
> > propagated to the storm workers, please correct me if that's wrong.
> >
> > But what if you make a change to the configs in the nimbus and then
> restart
> > it with the topology still running? i.e., will that change be propagated
> > to the existing topology workers somehow? (In the case of changing the
> > messaging transport, I assume that would mean restarting the workers to
> use
> > the alternative transport layer.)
> >
> > Thanks!
> >
> > - Erik
> >
> > P.S., I am well aware of the other stuff you need to do to get zmq
> working
> > in storm-0.9.1+; I just don't want to force all of our topologies to
> > specify the above configuration knob.
> >
> > P.P.S., I'm using storm-0.9.3
> >
>
Re: storm.yaml config: change on nimbus or worker machines?
Posted by Susheel Kumar Gadalay <sk...@gmail.com>.
If this is required for one topology better to set it in the Config object.
Config config = new Config();
config.put(Config.storm.messaging.transport: "backtype.storm.messaging.zmq");
On 12/5/14, Erik Weathers <ew...@groupon.com> wrote:
> Can someone please clarify for me how you properly specify worker-specific
> configuration in storm.yaml?
>
> e.g., I'm making a change to use zmq as the transport (due to a tangential
> problem with netty), and it's unclear where to make this change:
>
> - storm.messaging.transport: backtype.storm.messaging.zmq
>
> I assumed that I needed to change storm.yaml on the worker machine, but it
> seems that the change only shows up in the nimbus UI when I modify
> storm.yaml on the nimbus.
>
> So I'm assuming that the config options you specify in storm.yaml on nimbus
> propagated to the storm workers, please correct me if that's wrong.
>
> But what if you make a change to the configs in the nimbus and then restart
> it with the topology still running? i.e., will that change be propagated
> to the existing topology workers somehow? (In the case of changing the
> messaging transport, I assume that would mean restarting the workers to use
> the alternative transport layer.)
>
> Thanks!
>
> - Erik
>
> P.S., I am well aware of the other stuff you need to do to get zmq working
> in storm-0.9.1+; I just don't want to force all of our topologies to
> specify the above configuration knob.
>
> P.P.S., I'm using storm-0.9.3
>