You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Matthias J. Sax" <mj...@informatik.hu-berlin.de> on 2015/04/23 11:59:41 UTC
Why is Storm Config not modifiable?
Hi,
in the Spout/Bolt open()/prepare() method, a Map containing the current
configuration is given. This Map is of type
clojure.lang.PersistentHashMap and calling .put(...) raises an
UnsupportedOperationException.
I was wondering, why a persistent map is used? I could image, that
system defined values should not be modified. But adding new key/value
pairs is not allowed either (and would be very helpful in my case).
Right now, I simply copy all values into a new HashMap object and add my
own values to this new HashMap. But this is an annoying workaround I
have to do each time...
Would it make sense, the change the type of the system provided Map (or
modify it's behavior to all for adding new key/value pairs)?
-Matthias
Re: Why is Storm Config not modifiable?
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
The persistent map comes from clojure. Clojure is a functional language where all data types are immutable. My suggestion would be to keep doing what you are doing.
open(..., Map conf, ...) { conf = new HashMap(conf); ...
}
I know it can be a pain, but it also avoids situations like with Hadoop, where configs are modifiable and shared. I have had to fix race conditions because one part of the code is setting a config that it thinks no one else is using right now but another part of the code on a different thread is reading that same config, and sometimes it reads the old one, and other times it reads the new one. I really would rather have had an UnsupportedOperationException come out early on instead of trying to debug what was happening.
You could also create a few abstract parent classes that all of your bolts and spouts derive from.
... open(..., Map conf, ...) { modOpen(...,conf instanceof HashMap ? (HashMap)conf : new HashMap(conf), ...);
}
... abstract modOpen(..., HashMap conf, ...);
- Bobby
On Thursday, April 23, 2015 8:10 AM, Nathan Leung <nc...@gmail.com> wrote:
Instead of writing the map you can just write a variable in the abstract
class.
On Apr 23, 2015 7:45 AM, "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
wrote:
> I understand that.
>
> In my case, I use a class hierarchy with abstract spouts/bolts and
> multiple concrete spouts/bolts. The abstract class uses some default
> values (if not provided in the given config) similar to the derived
> classes. However, some derived classes overwrite the default values for
> the abstract class, ie, if no value is given in the config, some derived
> classes set the values and other do not. Additionally, open()/prepare()
> of the abstract class is called, too, and set the default value if the
> value was neither provided by the user, not by the derived class.
>
> Does it make sense how I explained it?
>
>
> -Matthias
>
>
>
>
> On 04/23/2015 01:08 PM, Nathan Leung wrote:
> > Config should be set when the topology is created before it's sent to the
> > nimbus. If you write an option to the map there is no mechanism to
> > propagate the change to the rest of the instances of storm components.
> > On Apr 23, 2015 6:02 AM, "Matthias J. Sax" <
> mjsax@informatik.hu-berlin.de>
> > wrote:
> >
> >> Hi,
> >>
> >> in the Spout/Bolt open()/prepare() method, a Map containing the current
> >> configuration is given. This Map is of type
> >> clojure.lang.PersistentHashMap and calling .put(...) raises an
> >> UnsupportedOperationException.
> >>
> >> I was wondering, why a persistent map is used? I could image, that
> >> system defined values should not be modified. But adding new key/value
> >> pairs is not allowed either (and would be very helpful in my case).
> >>
> >> Right now, I simply copy all values into a new HashMap object and add my
> >> own values to this new HashMap. But this is an annoying workaround I
> >> have to do each time...
> >>
> >> Would it make sense, the change the type of the system provided Map (or
> >> modify it's behavior to all for adding new key/value pairs)?
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >>
> >>
> >
>
>
Re: Why is Storm Config not modifiable?
Posted by Nathan Leung <nc...@gmail.com>.
Instead of writing the map you can just write a variable in the abstract
class.
On Apr 23, 2015 7:45 AM, "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
wrote:
> I understand that.
>
> In my case, I use a class hierarchy with abstract spouts/bolts and
> multiple concrete spouts/bolts. The abstract class uses some default
> values (if not provided in the given config) similar to the derived
> classes. However, some derived classes overwrite the default values for
> the abstract class, ie, if no value is given in the config, some derived
> classes set the values and other do not. Additionally, open()/prepare()
> of the abstract class is called, too, and set the default value if the
> value was neither provided by the user, not by the derived class.
>
> Does it make sense how I explained it?
>
>
> -Matthias
>
>
>
>
> On 04/23/2015 01:08 PM, Nathan Leung wrote:
> > Config should be set when the topology is created before it's sent to the
> > nimbus. If you write an option to the map there is no mechanism to
> > propagate the change to the rest of the instances of storm components.
> > On Apr 23, 2015 6:02 AM, "Matthias J. Sax" <
> mjsax@informatik.hu-berlin.de>
> > wrote:
> >
> >> Hi,
> >>
> >> in the Spout/Bolt open()/prepare() method, a Map containing the current
> >> configuration is given. This Map is of type
> >> clojure.lang.PersistentHashMap and calling .put(...) raises an
> >> UnsupportedOperationException.
> >>
> >> I was wondering, why a persistent map is used? I could image, that
> >> system defined values should not be modified. But adding new key/value
> >> pairs is not allowed either (and would be very helpful in my case).
> >>
> >> Right now, I simply copy all values into a new HashMap object and add my
> >> own values to this new HashMap. But this is an annoying workaround I
> >> have to do each time...
> >>
> >> Would it make sense, the change the type of the system provided Map (or
> >> modify it's behavior to all for adding new key/value pairs)?
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >>
> >>
> >
>
>
Re: Why is Storm Config not modifiable?
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
Thanks for your explanations.
IHMO, from a user point of view, the interface of open()/prepare() is
not well designed. It specifies to hand-in a Java-Map, and you would not
expect that a Map is not modifiable.
Would it be possible to change the parameter type to
clojure.lang.APersistentMap ? This would expose the persistent nature of
the map to the user. Not sure if this is possible, due to compatibility
issues... It's just a thought.
-Matthias
On 04/23/2015 06:10 PM, Bobby Evans wrote:
> Yes you could all assoc on the Map, if you cast the map to an IPersistentMap and call the assoc method on it, but if you do it blindly without checking they actual type of the Map passed in you may run into issues in the future, if we do change some internals away from java.
> - Bobby
>
>
>
> On Thursday, April 23, 2015 9:07 AM, Jeremy Heiler <je...@gmail.com> wrote:
>
>
> On Thu, Apr 23, 2015 at 4:42 AM, Matthias J. Sax <
> mjsax@informatik.hu-berlin.de> wrote:
>
>> In my case, I use a class hierarchy with abstract spouts/bolts and
>> multiple concrete spouts/bolts. The abstract class uses some default
>> values (if not provided in the given config) similar to the derived
>> classes. However, some derived classes overwrite the default values for
>> the abstract class, ie, if no value is given in the config, some derived
>> classes set the values and other do not. Additionally, open()/prepare()
>> of the abstract class is called, too, and set the default value if the
>> value was neither provided by the user, not by the derived class.
>>
>>
> The idiomatic way to update an immutable field like this in Clojure would
> be to use an atom. Here you could store the conf in an AtomicReference
> field, call .assoc on the map to update it, and .compareAndSet it on the
> AtomicReference. I'm not sure you'll need this, but I figured I'd mention
> it as how you would use a PersistentHashMap in your situation.
>
>
>
>
Re: Why is Storm Config not modifiable?
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Yes you could all assoc on the Map, if you cast the map to an IPersistentMap and call the assoc method on it, but if you do it blindly without checking they actual type of the Map passed in you may run into issues in the future, if we do change some internals away from java.
- Bobby
On Thursday, April 23, 2015 9:07 AM, Jeremy Heiler <je...@gmail.com> wrote:
On Thu, Apr 23, 2015 at 4:42 AM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> In my case, I use a class hierarchy with abstract spouts/bolts and
> multiple concrete spouts/bolts. The abstract class uses some default
> values (if not provided in the given config) similar to the derived
> classes. However, some derived classes overwrite the default values for
> the abstract class, ie, if no value is given in the config, some derived
> classes set the values and other do not. Additionally, open()/prepare()
> of the abstract class is called, too, and set the default value if the
> value was neither provided by the user, not by the derived class.
>
>
The idiomatic way to update an immutable field like this in Clojure would
be to use an atom. Here you could store the conf in an AtomicReference
field, call .assoc on the map to update it, and .compareAndSet it on the
AtomicReference. I'm not sure you'll need this, but I figured I'd mention
it as how you would use a PersistentHashMap in your situation.
Re: Why is Storm Config not modifiable?
Posted by Jeremy Heiler <je...@gmail.com>.
On Thu, Apr 23, 2015 at 4:42 AM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:
> In my case, I use a class hierarchy with abstract spouts/bolts and
> multiple concrete spouts/bolts. The abstract class uses some default
> values (if not provided in the given config) similar to the derived
> classes. However, some derived classes overwrite the default values for
> the abstract class, ie, if no value is given in the config, some derived
> classes set the values and other do not. Additionally, open()/prepare()
> of the abstract class is called, too, and set the default value if the
> value was neither provided by the user, not by the derived class.
>
>
The idiomatic way to update an immutable field like this in Clojure would
be to use an atom. Here you could store the conf in an AtomicReference
field, call .assoc on the map to update it, and .compareAndSet it on the
AtomicReference. I'm not sure you'll need this, but I figured I'd mention
it as how you would use a PersistentHashMap in your situation.
Re: Why is Storm Config not modifiable?
Posted by "Matthias J. Sax" <mj...@informatik.hu-berlin.de>.
I understand that.
In my case, I use a class hierarchy with abstract spouts/bolts and
multiple concrete spouts/bolts. The abstract class uses some default
values (if not provided in the given config) similar to the derived
classes. However, some derived classes overwrite the default values for
the abstract class, ie, if no value is given in the config, some derived
classes set the values and other do not. Additionally, open()/prepare()
of the abstract class is called, too, and set the default value if the
value was neither provided by the user, not by the derived class.
Does it make sense how I explained it?
-Matthias
On 04/23/2015 01:08 PM, Nathan Leung wrote:
> Config should be set when the topology is created before it's sent to the
> nimbus. If you write an option to the map there is no mechanism to
> propagate the change to the rest of the instances of storm components.
> On Apr 23, 2015 6:02 AM, "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
> wrote:
>
>> Hi,
>>
>> in the Spout/Bolt open()/prepare() method, a Map containing the current
>> configuration is given. This Map is of type
>> clojure.lang.PersistentHashMap and calling .put(...) raises an
>> UnsupportedOperationException.
>>
>> I was wondering, why a persistent map is used? I could image, that
>> system defined values should not be modified. But adding new key/value
>> pairs is not allowed either (and would be very helpful in my case).
>>
>> Right now, I simply copy all values into a new HashMap object and add my
>> own values to this new HashMap. But this is an annoying workaround I
>> have to do each time...
>>
>> Would it make sense, the change the type of the system provided Map (or
>> modify it's behavior to all for adding new key/value pairs)?
>>
>>
>> -Matthias
>>
>>
>>
>>
>>
>
Re: Why is Storm Config not modifiable?
Posted by Nathan Leung <nc...@gmail.com>.
Config should be set when the topology is created before it's sent to the
nimbus. If you write an option to the map there is no mechanism to
propagate the change to the rest of the instances of storm components.
On Apr 23, 2015 6:02 AM, "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
wrote:
> Hi,
>
> in the Spout/Bolt open()/prepare() method, a Map containing the current
> configuration is given. This Map is of type
> clojure.lang.PersistentHashMap and calling .put(...) raises an
> UnsupportedOperationException.
>
> I was wondering, why a persistent map is used? I could image, that
> system defined values should not be modified. But adding new key/value
> pairs is not allowed either (and would be very helpful in my case).
>
> Right now, I simply copy all values into a new HashMap object and add my
> own values to this new HashMap. But this is an annoying workaround I
> have to do each time...
>
> Would it make sense, the change the type of the system provided Map (or
> modify it's behavior to all for adding new key/value pairs)?
>
>
> -Matthias
>
>
>
>
>