You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Navin Ipe <na...@searchlighthealth.com> on 2016/04/19 11:22:31 UTC
How do you add a custom class to Config?
I have this
Config config = new Config();
MongoDatabaseManager mongoManager = new MongoDatabaseManager();
config.put("MongoManager", mongoManager);
and MongoDatabaseManager is an empty class:
public class MongoDatabaseManager implements Serializable {}
But after submitting the topology, I get this error:
*14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails - Creating a
new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with resources:
{"supervisor.cpu.capacity" 400.0, "supervisor.memory.capacity.mb"
3072.0}14:45:18.321 [main] ERROR o.a.s.s.o.a.z.s.NIOServerCnxnFactory -
Thread Thread[main,5,main] diedjava.lang.IllegalArgumentException: Topology
conf is not json-serializable at
org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
~[storm-core-1.0.0.jar:1.0.0] at
org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
~[storm-core-1.0.0.jar:1.0.0] at
org.apache.storm.LocalCluster.submitTopology(Unknown Source)
~[storm-core-1.0.0.jar:1.0.0] at
com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
Why can't I add a custom class to config? The idea here is to store some
info in the class so that all workers can access the information (because
the class is part of the topology) even though they are part of different
JVM's (the class can be accessed in the open() or prepare() functions of
spouts or bolts, but first, I don't understand why this crash is happening).
--
Regards,
Navin
Re: How do you add a custom class to Config?
Posted by Navin Ipe <na...@searchlighthealth.com>.
@Deepak: In the first email of this thread I had shown the code I was
trying to use. MongoDatabaseManager is serialized and I've even tried using
it as an empty class but it didn't work, which is why I asked on the
mailing list. Since it's a tiny piece of code, you could also try it out on
your system.
Config config = new Config();
MongoDatabaseManager mongoManager = new MongoDatabaseManager();
config.put("MongoManager", mongoManager);
and MongoDatabaseManager is an empty class:
public class MongoDatabaseManager implements Serializable {}
But after submitting the topology, I get this error:
*14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails - Creating a
new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with resources:
{"supervisor.cpu.capacity" 400.0, "supervisor.memory.capacity.mb"
3072.0}14:45:18.321 [main] ERROR o.a.s.s.o.a.z.s.NIOServerCnxnFactory -
Thread Thread[main,5,main] diedjava.lang.IllegalArgumentException: Topology
conf is not json-serializable at
org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
~[storm-core-1.0.0.jar:1.0.0] at
org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
~[storm-core-1.0.0.jar:1.0.0] at
org.apache.storm.LocalCluster.submitTopology(Unknown Source)
~[storm-core-1.0.0.jar:1.0.0] at
com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
On Thu, Apr 21, 2016 at 12:17 PM, Deepak Sharma <de...@gmail.com>
wrote:
> Hi Navin
> If the MongoDatabaseManager is serialized, you should be able to dd it to
> the config of the topology.
> If not , the class has to be serialized.
>
> Thanks
> Deepak
>
> On Thu, Apr 21, 2016 at 11:57 AM, Navin Ipe <
> navin.ipe@searchlighthealth.com> wrote:
>
>> Ah ok that's a nice idea, but in my situation it's like this:
>> I want to initialize a database connection and also hold some database
>> rows and other values in a class. Once these are initialized in a class, I
>> don't want any other class to initialize it for the sake of saving memory
>> and for referential integrity. It is this initialized object that I want
>> all my Spouts to be able to access.
>> That's why I was hoping that at least if Config could hold the class,
>> then all Spouts and Bolts in the topology would be able to access the
>> initialized values, because Config is part of the topology and can be
>> accessed in open() and prepare().
>>
>>
>> On Thu, Apr 21, 2016 at 11:08 AM, Abhishek Agarwal <ab...@gmail.com>
>> wrote:
>>
>>> In the prepare/open method, you can get the class name and initialize
>>> the instance from class name
>>>
>>> Excuse typos
>>> On Apr 21, 2016 9:17 AM, "Navin Ipe" <na...@searchlighthealth.com>
>>> wrote:
>>>
>>>> Thanks Abhishek, but I don't quite understand what you meant. Do you
>>>> mean I pass a class name of a static class and then use it?
>>>>
>>>> Also took a look at the source code and this is what it says:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> */** * Topology configs are specified as a plain old map. This class
>>>> provides a * convenient way to create a topology config map by providing
>>>> setter methods for * all the configs that can be set. It also makes it
>>>> easier to do things like add * serializations. * * This class also provides
>>>> constants for all the configurations possible on * a Storm cluster and
>>>> Storm topology. Each constant is paired with an annotation * that defines
>>>> the validity criterion of the corresponding field. Default * values for
>>>> these configs can be found in defaults.yaml. * * Note that you may put
>>>> other configurations in any of the configs. Storm * will ignore anything it
>>>> doesn't recognize, but your topologies are free to make * use of them by
>>>> reading them in the prepare method of Bolts or the open method of *
>>>> Spouts. */public class Config extends HashMap<String, Object> {*
>>>>
>>>> So if Storm is supposed to ignore anything it does not recognize, then
>>>> it's strange it's throwing an error for my serializable class.
>>>>
>>>> On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <
>>>> abhishcool@gmail.com> wrote:
>>>>
>>>>> Pass the class name in config instead of concrete object.
>>>>>
>>>>> Excuse typos
>>>>> On Apr 20, 2016 9:43 PM, "Navin Ipe" <na...@searchlighthealth.com>
>>>>> wrote:
>>>>>
>>>>>> Does anyone know how to do this? If not adding a custom class, then
>>>>>> at least about how to maintain information retrieved from a database in a
>>>>>> single class which all spouts/bolts in all the workers will be able to
>>>>>> access?
>>>>>>
>>>>>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
>>>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>>>
>>>>>>> I have this
>>>>>>>
>>>>>>> Config config = new Config();
>>>>>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>>>>>>> config.put("MongoManager", mongoManager);
>>>>>>>
>>>>>>> and MongoDatabaseManager is an empty class:
>>>>>>> public class MongoDatabaseManager implements Serializable {}
>>>>>>>
>>>>>>> But after submitting the topology, I get this error:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails -
>>>>>>> Creating a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with
>>>>>>> resources: {"supervisor.cpu.capacity" 400.0,
>>>>>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>>>>>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>>>>>>> diedjava.lang.IllegalArgumentException: Topology conf is not
>>>>>>> json-serializable at
>>>>>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>>>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>>>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>>>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>>>>>>
>>>>>>> Why can't I add a custom class to config? The idea here is to store
>>>>>>> some info in the class so that all workers can access the information
>>>>>>> (because the class is part of the topology) even though they are part of
>>>>>>> different JVM's (the class can be accessed in the open() or prepare()
>>>>>>> functions of spouts or bolts, but first, I don't understand why this crash
>>>>>>> is happening).
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Navin
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Navin
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Navin
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Navin
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>
--
Regards,
Navin
Re: How do you add a custom class to Config?
Posted by Deepak Sharma <de...@gmail.com>.
Hi Navin
If the MongoDatabaseManager is serialized, you should be able to dd it to
the config of the topology.
If not , the class has to be serialized.
Thanks
Deepak
On Thu, Apr 21, 2016 at 11:57 AM, Navin Ipe <navin.ipe@searchlighthealth.com
> wrote:
> Ah ok that's a nice idea, but in my situation it's like this:
> I want to initialize a database connection and also hold some database
> rows and other values in a class. Once these are initialized in a class, I
> don't want any other class to initialize it for the sake of saving memory
> and for referential integrity. It is this initialized object that I want
> all my Spouts to be able to access.
> That's why I was hoping that at least if Config could hold the class, then
> all Spouts and Bolts in the topology would be able to access the
> initialized values, because Config is part of the topology and can be
> accessed in open() and prepare().
>
>
> On Thu, Apr 21, 2016 at 11:08 AM, Abhishek Agarwal <ab...@gmail.com>
> wrote:
>
>> In the prepare/open method, you can get the class name and initialize the
>> instance from class name
>>
>> Excuse typos
>> On Apr 21, 2016 9:17 AM, "Navin Ipe" <na...@searchlighthealth.com>
>> wrote:
>>
>>> Thanks Abhishek, but I don't quite understand what you meant. Do you
>>> mean I pass a class name of a static class and then use it?
>>>
>>> Also took a look at the source code and this is what it says:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> */** * Topology configs are specified as a plain old map. This class
>>> provides a * convenient way to create a topology config map by providing
>>> setter methods for * all the configs that can be set. It also makes it
>>> easier to do things like add * serializations. * * This class also provides
>>> constants for all the configurations possible on * a Storm cluster and
>>> Storm topology. Each constant is paired with an annotation * that defines
>>> the validity criterion of the corresponding field. Default * values for
>>> these configs can be found in defaults.yaml. * * Note that you may put
>>> other configurations in any of the configs. Storm * will ignore anything it
>>> doesn't recognize, but your topologies are free to make * use of them by
>>> reading them in the prepare method of Bolts or the open method of *
>>> Spouts. */public class Config extends HashMap<String, Object> {*
>>>
>>> So if Storm is supposed to ignore anything it does not recognize, then
>>> it's strange it's throwing an error for my serializable class.
>>>
>>> On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <abhishcool@gmail.com
>>> > wrote:
>>>
>>>> Pass the class name in config instead of concrete object.
>>>>
>>>> Excuse typos
>>>> On Apr 20, 2016 9:43 PM, "Navin Ipe" <na...@searchlighthealth.com>
>>>> wrote:
>>>>
>>>>> Does anyone know how to do this? If not adding a custom class, then at
>>>>> least about how to maintain information retrieved from a database in a
>>>>> single class which all spouts/bolts in all the workers will be able to
>>>>> access?
>>>>>
>>>>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
>>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>>
>>>>>> I have this
>>>>>>
>>>>>> Config config = new Config();
>>>>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>>>>>> config.put("MongoManager", mongoManager);
>>>>>>
>>>>>> and MongoDatabaseManager is an empty class:
>>>>>> public class MongoDatabaseManager implements Serializable {}
>>>>>>
>>>>>> But after submitting the topology, I get this error:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails -
>>>>>> Creating a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with
>>>>>> resources: {"supervisor.cpu.capacity" 400.0,
>>>>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>>>>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>>>>>> diedjava.lang.IllegalArgumentException: Topology conf is not
>>>>>> json-serializable at
>>>>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>>>>>
>>>>>> Why can't I add a custom class to config? The idea here is to store
>>>>>> some info in the class so that all workers can access the information
>>>>>> (because the class is part of the topology) even though they are part of
>>>>>> different JVM's (the class can be accessed in the open() or prepare()
>>>>>> functions of spouts or bolts, but first, I don't understand why this crash
>>>>>> is happening).
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Navin
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Navin
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>
>
> --
> Regards,
> Navin
>
--
Thanks
Deepak
www.bigdatabig.com
www.keosha.net
Re: How do you add a custom class to Config?
Posted by Navin Ipe <na...@searchlighthealth.com>.
Ah ok that's a nice idea, but in my situation it's like this:
I want to initialize a database connection and also hold some database rows
and other values in a class. Once these are initialized in a class, I don't
want any other class to initialize it for the sake of saving memory and for
referential integrity. It is this initialized object that I want all my
Spouts to be able to access.
That's why I was hoping that at least if Config could hold the class, then
all Spouts and Bolts in the topology would be able to access the
initialized values, because Config is part of the topology and can be
accessed in open() and prepare().
On Thu, Apr 21, 2016 at 11:08 AM, Abhishek Agarwal <ab...@gmail.com>
wrote:
> In the prepare/open method, you can get the class name and initialize the
> instance from class name
>
> Excuse typos
> On Apr 21, 2016 9:17 AM, "Navin Ipe" <na...@searchlighthealth.com>
> wrote:
>
>> Thanks Abhishek, but I don't quite understand what you meant. Do you mean
>> I pass a class name of a static class and then use it?
>>
>> Also took a look at the source code and this is what it says:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> */** * Topology configs are specified as a plain old map. This class
>> provides a * convenient way to create a topology config map by providing
>> setter methods for * all the configs that can be set. It also makes it
>> easier to do things like add * serializations. * * This class also provides
>> constants for all the configurations possible on * a Storm cluster and
>> Storm topology. Each constant is paired with an annotation * that defines
>> the validity criterion of the corresponding field. Default * values for
>> these configs can be found in defaults.yaml. * * Note that you may put
>> other configurations in any of the configs. Storm * will ignore anything it
>> doesn't recognize, but your topologies are free to make * use of them by
>> reading them in the prepare method of Bolts or the open method of *
>> Spouts. */public class Config extends HashMap<String, Object> {*
>>
>> So if Storm is supposed to ignore anything it does not recognize, then
>> it's strange it's throwing an error for my serializable class.
>>
>> On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <ab...@gmail.com>
>> wrote:
>>
>>> Pass the class name in config instead of concrete object.
>>>
>>> Excuse typos
>>> On Apr 20, 2016 9:43 PM, "Navin Ipe" <na...@searchlighthealth.com>
>>> wrote:
>>>
>>>> Does anyone know how to do this? If not adding a custom class, then at
>>>> least about how to maintain information retrieved from a database in a
>>>> single class which all spouts/bolts in all the workers will be able to
>>>> access?
>>>>
>>>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
>>>> navin.ipe@searchlighthealth.com> wrote:
>>>>
>>>>> I have this
>>>>>
>>>>> Config config = new Config();
>>>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>>>>> config.put("MongoManager", mongoManager);
>>>>>
>>>>> and MongoDatabaseManager is an empty class:
>>>>> public class MongoDatabaseManager implements Serializable {}
>>>>>
>>>>> But after submitting the topology, I get this error:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails -
>>>>> Creating a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with
>>>>> resources: {"supervisor.cpu.capacity" 400.0,
>>>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>>>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>>>>> diedjava.lang.IllegalArgumentException: Topology conf is not
>>>>> json-serializable at
>>>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>>>>
>>>>> Why can't I add a custom class to config? The idea here is to store
>>>>> some info in the class so that all workers can access the information
>>>>> (because the class is part of the topology) even though they are part of
>>>>> different JVM's (the class can be accessed in the open() or prepare()
>>>>> functions of spouts or bolts, but first, I don't understand why this crash
>>>>> is happening).
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Navin
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Navin
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Navin
>>
>
--
Regards,
Navin
Re: How do you add a custom class to Config?
Posted by Abhishek Agarwal <ab...@gmail.com>.
In the prepare/open method, you can get the class name and initialize the
instance from class name
Excuse typos
On Apr 21, 2016 9:17 AM, "Navin Ipe" <na...@searchlighthealth.com>
wrote:
> Thanks Abhishek, but I don't quite understand what you meant. Do you mean
> I pass a class name of a static class and then use it?
>
> Also took a look at the source code and this is what it says:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> */** * Topology configs are specified as a plain old map. This class
> provides a * convenient way to create a topology config map by providing
> setter methods for * all the configs that can be set. It also makes it
> easier to do things like add * serializations. * * This class also provides
> constants for all the configurations possible on * a Storm cluster and
> Storm topology. Each constant is paired with an annotation * that defines
> the validity criterion of the corresponding field. Default * values for
> these configs can be found in defaults.yaml. * * Note that you may put
> other configurations in any of the configs. Storm * will ignore anything it
> doesn't recognize, but your topologies are free to make * use of them by
> reading them in the prepare method of Bolts or the open method of *
> Spouts. */public class Config extends HashMap<String, Object> {*
>
> So if Storm is supposed to ignore anything it does not recognize, then
> it's strange it's throwing an error for my serializable class.
>
> On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <ab...@gmail.com>
> wrote:
>
>> Pass the class name in config instead of concrete object.
>>
>> Excuse typos
>> On Apr 20, 2016 9:43 PM, "Navin Ipe" <na...@searchlighthealth.com>
>> wrote:
>>
>>> Does anyone know how to do this? If not adding a custom class, then at
>>> least about how to maintain information retrieved from a database in a
>>> single class which all spouts/bolts in all the workers will be able to
>>> access?
>>>
>>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
>>> navin.ipe@searchlighthealth.com> wrote:
>>>
>>>> I have this
>>>>
>>>> Config config = new Config();
>>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>>>> config.put("MongoManager", mongoManager);
>>>>
>>>> and MongoDatabaseManager is an empty class:
>>>> public class MongoDatabaseManager implements Serializable {}
>>>>
>>>> But after submitting the topology, I get this error:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails -
>>>> Creating a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with
>>>> resources: {"supervisor.cpu.capacity" 400.0,
>>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>>>> diedjava.lang.IllegalArgumentException: Topology conf is not
>>>> json-serializable at
>>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>>>
>>>> Why can't I add a custom class to config? The idea here is to store
>>>> some info in the class so that all workers can access the information
>>>> (because the class is part of the topology) even though they are part of
>>>> different JVM's (the class can be accessed in the open() or prepare()
>>>> functions of spouts or bolts, but first, I don't understand why this crash
>>>> is happening).
>>>>
>>>> --
>>>> Regards,
>>>> Navin
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>
>
> --
> Regards,
> Navin
>
Re: How do you add a custom class to Config?
Posted by Navin Ipe <na...@searchlighthealth.com>.
Thanks Abhishek, but I don't quite understand what you meant. Do you mean I
pass a class name of a static class and then use it?
Also took a look at the source code and this is what it says:
*/** * Topology configs are specified as a plain old map. This class
provides a * convenient way to create a topology config map by providing
setter methods for * all the configs that can be set. It also makes it
easier to do things like add * serializations. * * This class also provides
constants for all the configurations possible on * a Storm cluster and
Storm topology. Each constant is paired with an annotation * that defines
the validity criterion of the corresponding field. Default * values for
these configs can be found in defaults.yaml. * * Note that you may put
other configurations in any of the configs. Storm * will ignore anything it
doesn't recognize, but your topologies are free to make * use of them by
reading them in the prepare method of Bolts or the open method of *
Spouts. */public class Config extends HashMap<String, Object> {*
So if Storm is supposed to ignore anything it does not recognize, then it's
strange it's throwing an error for my serializable class.
On Thu, Apr 21, 2016 at 12:01 AM, Abhishek Agarwal <ab...@gmail.com>
wrote:
> Pass the class name in config instead of concrete object.
>
> Excuse typos
> On Apr 20, 2016 9:43 PM, "Navin Ipe" <na...@searchlighthealth.com>
> wrote:
>
>> Does anyone know how to do this? If not adding a custom class, then at
>> least about how to maintain information retrieved from a database in a
>> single class which all spouts/bolts in all the workers will be able to
>> access?
>>
>> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
>> navin.ipe@searchlighthealth.com> wrote:
>>
>>> I have this
>>>
>>> Config config = new Config();
>>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>>> config.put("MongoManager", mongoManager);
>>>
>>> and MongoDatabaseManager is an empty class:
>>> public class MongoDatabaseManager implements Serializable {}
>>>
>>> But after submitting the topology, I get this error:
>>>
>>>
>>>
>>>
>>>
>>>
>>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails - Creating
>>> a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with
>>> resources: {"supervisor.cpu.capacity" 400.0,
>>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>>> diedjava.lang.IllegalArgumentException: Topology conf is not
>>> json-serializable at
>>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>>> ~[storm-core-1.0.0.jar:1.0.0] at
>>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>>
>>> Why can't I add a custom class to config? The idea here is to store some
>>> info in the class so that all workers can access the information (because
>>> the class is part of the topology) even though they are part of different
>>> JVM's (the class can be accessed in the open() or prepare() functions of
>>> spouts or bolts, but first, I don't understand why this crash is happening).
>>>
>>> --
>>> Regards,
>>> Navin
>>>
>>
>>
>>
>> --
>> Regards,
>> Navin
>>
>
--
Regards,
Navin
Re: How do you add a custom class to Config?
Posted by Abhishek Agarwal <ab...@gmail.com>.
Pass the class name in config instead of concrete object.
Excuse typos
On Apr 20, 2016 9:43 PM, "Navin Ipe" <na...@searchlighthealth.com>
wrote:
> Does anyone know how to do this? If not adding a custom class, then at
> least about how to maintain information retrieved from a database in a
> single class which all spouts/bolts in all the workers will be able to
> access?
>
> On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <
> navin.ipe@searchlighthealth.com> wrote:
>
>> I have this
>>
>> Config config = new Config();
>> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
>> config.put("MongoManager", mongoManager);
>>
>> and MongoDatabaseManager is an empty class:
>> public class MongoDatabaseManager implements Serializable {}
>>
>> But after submitting the topology, I get this error:
>>
>>
>>
>>
>>
>>
>> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails - Creating
>> a new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with
>> resources: {"supervisor.cpu.capacity" 400.0,
>> "supervisor.memory.capacity.mb" 3072.0}14:45:18.321 [main] ERROR
>> o.a.s.s.o.a.z.s.NIOServerCnxnFactory - Thread Thread[main,5,main]
>> diedjava.lang.IllegalArgumentException: Topology conf is not
>> json-serializable at
>> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
>> ~[storm-core-1.0.0.jar:1.0.0] at
>> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
>> ~[storm-core-1.0.0.jar:1.0.0] at
>> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
>> ~[storm-core-1.0.0.jar:1.0.0] at
>> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>>
>> Why can't I add a custom class to config? The idea here is to store some
>> info in the class so that all workers can access the information (because
>> the class is part of the topology) even though they are part of different
>> JVM's (the class can be accessed in the open() or prepare() functions of
>> spouts or bolts, but first, I don't understand why this crash is happening).
>>
>> --
>> Regards,
>> Navin
>>
>
>
>
> --
> Regards,
> Navin
>
Re: How do you add a custom class to Config?
Posted by Navin Ipe <na...@searchlighthealth.com>.
Does anyone know how to do this? If not adding a custom class, then at
least about how to maintain information retrieved from a database in a
single class which all spouts/bolts in all the workers will be able to
access?
On Tue, Apr 19, 2016 at 2:52 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:
> I have this
>
> Config config = new Config();
> MongoDatabaseManager mongoManager = new MongoDatabaseManager();
> config.put("MongoManager", mongoManager);
>
> and MongoDatabaseManager is an empty class:
> public class MongoDatabaseManager implements Serializable {}
>
> But after submitting the topology, I get this error:
>
>
>
>
>
>
> *14:45:18.320 [timer] DEBUG o.a.s.scheduler.SupervisorDetails - Creating a
> new supervisor (null-5b950364-129e-4e04-b10c-d1e496c12043) with resources:
> {"supervisor.cpu.capacity" 400.0, "supervisor.memory.capacity.mb"
> 3072.0}14:45:18.321 [main] ERROR o.a.s.s.o.a.z.s.NIOServerCnxnFactory -
> Thread Thread[main,5,main] diedjava.lang.IllegalArgumentException: Topology
> conf is not json-serializable at
> org.apache.storm.testing$submit_local_topology.invoke(testing.clj:299)
> ~[storm-core-1.0.0.jar:1.0.0] at
> org.apache.storm.LocalCluster$_submitTopology.invoke(LocalCluster.clj:49)
> ~[storm-core-1.0.0.jar:1.0.0] at
> org.apache.storm.LocalCluster.submitTopology(Unknown Source)
> ~[storm-core-1.0.0.jar:1.0.0] at
> com.slh.Basicstorm.BasicStorm.main(BasicStorm.java:44) ~[main/:na]*
>
> Why can't I add a custom class to config? The idea here is to store some
> info in the class so that all workers can access the information (because
> the class is part of the topology) even though they are part of different
> JVM's (the class can be accessed in the open() or prepare() functions of
> spouts or bolts, but first, I don't understand why this crash is happening).
>
> --
> Regards,
> Navin
>
--
Regards,
Navin